GSoC 2017 - p5.js Native

palmerpa

p5.js Native

Synopsis

The goal of the p5.js Native project is to develop a tool that can create simple, cross-platform, standalone mobile applications from p5.js sketches. The supported platforms will be iOS, Android, and potentially Windows Phone. The target user base for the tool is novice to intermediate developers and designers. In order to reach this audience, the tool will require minimal configuration on the part of the developer. The implementation of p5.js Native will likely depend on Apache Cordova for wrapping the sketch in a WebView and exposing the OS APIs to the sketch.

Benefits to Community

Currently, p5.js sketches are executed in a mobile web browser. One problem with this model is that not all browsers behave consistently (particularly in regards to experimental Web APIs) which can make for an unpleasant experience for the end user. There are many open GitHub issues in the p5.js repo describing strange bugs related to OS/browser behavior, see #884, #1313, #1672, #1847. By running p5.js apps in a native environment many of the browser bugs can be completely avoided.
Furthermore, running code in the browser restricts p5.js sketches to only sensors that are supported by Web APIs, some of which are either experimental or have not yet been standardized. p5.js Native will use Cordova plugins to expose a whole slew of APIs that are normally only accessible to native applications (see the core Cordova plugins for a sampling of the APIs that can be supported). Giving developers access to more mobile device APIs will allow them to create richer mobile applications.

Deliverables

A production-ready tool for creating cross-platform mobile applications from p5.js sketches.
User-friendly documentation and/or tutorials about how to use the p5.js Native tool.

Biographical Information

Two years ago, I created the Processing Mobile library that exposed touch controls and mobile device sensors to Processing sketches so that developers could easily extend their creations to the mobile platform. (Nearly all of the library’s functionality has been essentially subsumed under p5.js at this point.) p5.js Native would greatly expand upon the Processing Mobile project by making it possible for developers and designers to create entirely native mobile experiences using only p5.js.
Additionally, I have extensive experience with Processing. I have used it for numerous projects over the past 3+ years, including an interactive art exhibit. I understand the Processing community, the goals of the Processing Foundation, and the role that Processing serves for developers and designers. These attributes make me confident that I can successfully complete the p5.js Native project.

Contact Information

Name: Palmer Paul
Website: palmerpaul.com
GitHub: pzp1997
IRC (freenode): pzp
Please contact me via email for my telephone number and postal address

palmerpa

Looking for feedback on my project idea. All advice, comments, etc. are welcome.

shiffman

Thanks for this proposal, I love the idea of expanding the possibilities for creative mobile development with p5. One thing I have noticed in my teaching is that a main barrier is simply the workflow and debugging. How to write the code on a laptop but deploy to mobile, get console logging to laptop, and be able to quickly iterate and make changes. Do you think this project can help to address mobile development workflow complications?

palmerpa

Thank you for your feedback @shiffman. I agree with you that the mobile development workflow could seem daunting to beginners. The primary consideration of p5.js Native would be to make the process as simple and straight-forward as possible. I have a few ideas that address your comments.

Debugging

As an initial step, p5.js Native would only support the core p5.js API. At that stage, debugging would be pretty easy as all p5.js Native sketches are valid p5.js sketches. This means that sketches can be developed and tested in the usual way and then transformed into a mobile app using p5.js Native where in theory they should behave in the same way. The normal p5.js debugging workflow will probably continue to work even after some of the more popular libraries, like p5.dom and p5.sound, are introduced to p5.js Native as the sketches are still running in just a WebView.

From what I see, the normal debugging workflow breaks down once p5.js Native begins supporting APIs that are exclusive to mobile platforms. When that happens, I believe Firebase Analytics or some other analytics service could be useful for more advanced debugging. Firebase Analytics offers a comprehensive logging service that can be viewed from the Firebase web portal. I think the best way to integrate it in a way that is transparent to p5.js Native users is to override console.log (and other related console functions) to redirect messages to Firebase. An added benefit to using Firebase Analytics, is that it can also keep track of crash logs and other important statistics like user demographics. My initial thoughts are to make Firebase integration an opt-in feature, as it would require p5.js Native users to create a Firebase project from the Firebase console, and we want to keep the process as simple as possible.

Development (re: quick design iteration)

An interesting idea to mostly remove the need to re-deploy during development would be to host the source code for the sketch online (we might need to disable resource caching in Cordova to get this to work). That way the Cordova app can just reload the resource from the web without the entire app needing to be re-built and re-deployed to the testing device. I believe React Native does something like this for development by packaging up the resources for the app into a bundle and hosting it on a local server that can be accessed from the app. I believe that we should shy away from using this practice in production though.

Deployment

Deploying to mobile devices is a little bit tricky. Cordova provides a CLI tool for deploying to iOS devices and running simulators to test apps on your computer. Installing the stuff to make this work is not a very difficult process but I would not consider it trivial either. I would need to do more research to see how much of the process can be automated behind the scenes using p5.js Native. I will also try to find an existing GUI for Cordova, which I believe could make the process much more user-friendly.

Hopefully, this addresses some of your concerns about the workflow. I believe that out of the three areas, deployment will require the most effort to perfect, although I am confident that the process can be simplified somewhat. Thanks again for your feedback. Please let me know if you have any more advice for me.

lmccart

hey @palmerpa, interesting idea. it might be worth taking a look through this repository which was a gsoc project from 2015 that didn't quite get released: https://github.com/OhJia/p5-mobile-phonegap

I agree that thinking about and refining the workflow will be an important part of this project.

palmerpa

Thank you @lmccart for informing me about that project. I have read through its associated wiki and glanced at some of the source code. Would you happen to have any more information about it, such as what technical obstacles prevented it from being completed?

lmccart

I don't have much more info... you could try leaving a question as an issue on the repo. I believe the main obstacle was just lack of time as the student was in the middle of masters thesis at the same time.

palmerpa

Submitted a draft of a proposal on the GSoC website. In case those who do not have access to the submitted drafts would like to take a look, the link to it is... https://docs.google.com/document/d/1FNkbLOix-E8eGakyDCfjTOfAhIGWXbk1flDRV3QX3lA/edit. All comments are welcome!