Yeaaaah I can’t talk about that. Dang, it was really cool though!

Yeaaaah I can’t talk about that. Dang, it was really cool though!

MINDSEED

Abstract

Mindseed is an experiment that uses virtual reality and machine learning tools to facilitate the generation of new ideas through visualization, language, immersive experience, and play. I see the future of the project as a digital framework that uses virtual reality, a web platform, and natural language processing to create a world where you can plant, grow, and harvest your ideas.

You can watch a video of the project presentation at ITP’s thesis week below.

Thesis Presentation Video:

TECHNICAL DETAILS

Hardware: PC, Oculus Quest 2, HTC Vive
Software: Unreal Engine 4, React, ThreeJS, Python, Heroku, Tensorflow, MongoDB, Spacy
Development time: 4 Months

Introduction

For my thesis, I built a prototype to assist myself with my creative process. We all know what a struggle it is to come up with ideas. They take time, patience, and inspiration. Imagine if we could make the process of coming up with an idea more entertaining and engaging. What if we could find a way to grow ideas as if we were growing plants or trees. Through these past 14 weeks, I’ve found a way to prototype this notion. Mindseed is a digital framework that uses virtual reality, a web platform, and natural language processing to create a world where you can plant, grow, and harvest your ideas.

Personal Statement

 So, I’ve been saving ideas for a while. I started using online platforms to record them in 2017, including google drive, keep, notes, and even emails, slowly building a personal archive of my thoughts. Since then, I’ve relied on this archive to develop new projects and recently, to think about my thesis project. I started my thesis exploration by frequently reviewing this archive and adding new ideas to it. 

After some time, I realized many of these concepts were connected, and some became much more interesting when I mixed parts and pieces of them together.I enjoyed this process of mixing concepts, and I was excited by some of the results. so I asked myself,  How I can translate this experiment with language into a digital experience? How can it benefit our creative process?

Research

Creativity is one of our most valuable skills, and our creative process is an essential part of innovation. This interest in innovation and problem-solving has led to the development of ideation methods and techniques like brainstorming and cheatstorming; It has fueled the success of collaboration tools such as Miro and inspired experimentation with online sites like Ideas.AI.Tools like these directly impact how we process information, engage with our ideas, and structure our creative process. 

Until recently our handheld screens and computer monitors have been the most utilized medium to interact with our data. As valuable as they can be, sometimes we feel detached from our digital information because it seems disembodied and shapeless. Today, we have new ways to visualize and experience our data. With the rise of technologies like mixed reality and machine learning, we can think of creating experiences that allow a more profound sense of embodiment, user personalization, and immersion.

I chose virtual reality to create an immersive experience with a more embodied sense of interaction. Inside this digital environment, we can experiment with any audiovisual representation to give life to our visual metaphors. On the other hand, using Natural language processing we have created tools that use semantic analysis and text generation to create meaningful communications that sometimes appear human. Mindseed is an experiment that uses virtual reality and machine learning tools to facilitate the generation of new ideas, using data visualization, language, immersive experience, and play.

Design / Production / Implementation

 The project has a simple premise: an idea is a seed. You plant your seed through a website by typing in a sentence that summarizes it. For example, “A website that helps people better understand their personal data. The website takes care of planting your seed and gives you back a set of coordinates indicating where you can find it later, once it has had the chance to grow. 

You then go into the VR world, where you enter the “Forest of Ideas.” Each of the ideas you planted earlier has become a full-grown fruit tree, with each fruit as a concept that makes up your idea. You can find the tree you planted with the coordinates you were given earlier, or you can walk around the forest and explore. 

 You go around the forest, picking around fruits from the ideas you like. When you feel you have enough, you mix them up and get a new seed to plant. This unique idea inherits some of the concepts you used to make it and will grow into an entire idea after some time.

You’ll receive a text message later on confirming when your new seedling has grown into a tree, as well as its coordinates and an id number for you to track it later.

Since this is all connected to a database, you’ll be able to mix different generations of ideas and track how they are related to each other. In further iterations, this will allow you to create a map of the roots that connect your thoughts.

Through my research, I found creative people prefer to generate ideas in solitary and later share these ideas with others. This is why I decided to incorporate multiplayer into the VR experience, allowing users to share limited interaction in the same space. Mindseed allows for individual reflection and also for cooperative play.

User Experience

There is a website where people can plant their ideas. It is currently functional, hosted online. The interface is simple; you type in your idea, choose a color for your seed, and then watch an animation of your seed being planted. You will also receive a set of coordinates. These are the coordinates of your tree inside the VR world.

The forest experience, where we can interact with our ideas, is made for VR. Additionally, it is also accessible to people who don’t have a VR headset and use a computer. It consists of two parts: A tutorial where someone can become familiarized with the controls, the environment, and how to interact with their ideas. The second part is the Forest, where you can explore alone or with others.

It took 14 weeks to develop Mindseed. The framework consists of three parts. The Frontend is the website where users can type in their ideas and see their seeds grow. The Backend takes care of processing the information, serving as a bridge between the website and the VR forest, making the experience of planting an idea seamless and synchronized. It also makes the simulated world seem continuously evolving, keeping track of each idea’s growth process.

User Experience

Onboarding

Mindseed has an onboarding experience that consists of a lobby where the user can learn how to move around, use the controls, and learn the dynamics of how to grow an idea. You can see a short video below.

Web Platform

Multiplayer VR

Text message notification

Multiplayer VR

Some results from the more fine-tuned algorithm:

• A project that uses random oscillating patterns to create frame-by-frame animations in 3D.
• A custom built sound controller that uses wind energy to mine cryptocurrency.
• A permanent installation for Berlin’s Futurism exhibition that explores prized possession in their native habitat.
• An interacive installation that facilitates collabortation between a human and a puddle of alcohol.
• A dancer’s body is extended and manipulated as a tool to quantify the world.

Conclusions and Next Steps

The process took several iterations, user testing, breaking, and building.

And it was the process that truly had me excited. Figuring out how to use these tools to generate unexpected text was surprisingly fun. Playing with all these VR dynamics, introducing smoke, and light truly felt magical. And seeing the project evolve from something that looked like a rough idea I wrote down into something so personal and relatable, makes me want to develop it even further into the future.

1. BE CREATIVE ALONE. BE CREATIVE TOGETHER – My research has led me to believe that coming up with ideas benefit from a solitary process. However, collective and constructive feedback can make a seed grow and evolve into true innovation.
2. TIME AS A BRIDGE – While we strive for seamless real-time communication, we sometimes forget the value of time as an essential part of our processes. Time is not something we should eliminate, but rather something we should work with to give our metaphors more power.
3. GO FOR THE LOW HANGING FRUIT – Our world is full of sensors and information. Make the most out of the information you have available to make the experience evolve with the person using it.
4. LANGUAGE IS PLAY
   Our language determines how we think, how we communicate, and what we create. We should think of language as a playful artifact, a poetic machine, and the center of our creative ideation process.
5. WE CHANGE. OUR IDEAS CHANGE.  We should be able to track these ideas and connect them with our past thoughts.  What better way to connect with ourselves than collaborating with our past to build our future!

For the next steps, I want to make the forest something you can mold into your own, a memory palace where you can freely explore your ideas and organize them spatially. I also believe each tree and fruit should look unique and recognizable, similar to the idea of a phenotype that expresses an inner trait. Appearance should have meaning. Through this process, several people have expressed their interest in using Mindseed. I want to further develop this into an experiment that can be experienced by more people.

Luminaria

---

Hardware: Oculus Quest 2
Software: Unity3D, Python, C#
Development time: 3 days

---

Luminaria is a playful experiment that uses several natural language processing techniques to generate random combination of ideas related to the art and tech world. I used a python script to scrape every blog post on the Creative Applications Network website and over 3000 tweets from the amazing MagicRealismBot in order to use Markov chain algorithms and Tracery to forge new interesting combinations of text. You can read more about the process, including code and examples below.

.

Process

Creative Applications Network

You can follow along with this notebook I made in Google Collab. I decided to gather all of the blog entry abstracts from the blog. You can identify them in the image below as the short abstracts of text you see below each title. After scraping the 147 entries, I made a CSV file out of them so I could take a better look.

I noticed most of the entries (or at least the ones I was interested in) followed a similar pattern. “X Name” is a “Y adjective” “Z New Media project”. I made a small diagram of it below.

Right away I thought this type of one-line format (after cleaning up the data a bit) would mix perfectly with Magic Realism Bot’s posts. So I took all of those and formatted them into a format acceptable by Spacy and to be processed with the Markov chain algorithm. I then proceeded to scrape tweets from the MagicRealismBot on Twitter.

Magic Realism Bot

I will not go into much detail about how I scraped it, since it is commented in the notebook. I obtained about 3200 tweets, which I took a screenshot of below.

Spacy, Tracery and the fun part

After having both my sources, I joined them into one CSV file. I processed the file using Spacy and Allison Parrish’s guide for corpus-driven narrative generation. The whole collection was about 4800 entries.

I also used a copy of Allison’s notebook. Using Spacy I identified entities, as well as actions, verbs, objects, and other linguistic items, which I used as input for Tracery.

I made a couple of tests to see how it would go. You can see some of the results below

You can see some of my favorites below:

“An interactive installation that facilitates collaboration between a human and a puddle of alcohol.”

Other remarkable examples:

A bisexual fisherman falls in love with the use of CCTV.

A professor reads a poem about an arduino that can destroy metaphysics.

A new life as a performance.

A theologian discovers that Wikipedia does not exist.

The dancer’s body is extended and manipulated as a tool to quantify the world.

An interactive installation and performance inspired by light rays traveling in a latent space of situations.

A project explores possible alternatives of how we experience the materiality of nature through the mediums of fiction.

Exploring behavior-based design systems that are self-aware, mobile, and self-structure / assemble.

To compensate for the lack of material I had from the Blog, I had to duplicate those entries several times, as well as some of the examples shown before. By the end it came out to a 3200(MRbot) vs. 1600 entries.(creativeappsnet + handpicked generated).

Output?

I came around this talk by Kate Compton (creator of Tracery) who lead the most enlightening discussion about procedural generation, and it dawned on me that I did not want to leave the output of this experiment only in text.

Through this VR experiment, I included several elements I became captivated by earlier this year. Such as the sense of waiting present in Epitaph, or the idea of spatial and environmental storytelling in Bitsy. I felt some of these generated ideas were brilliant in their own way, and deserved more attention that could not be afforded through a computer screen.

Ideally, each of these beams of light generate a set of ideas that vary in parameter values. Ideally, all of them could span a large area of the generation latent space. You would be able to walk to other beams of light and pick up some of these and keep them.

You can see how the quick prototype looked below.

Horizonte Clock

---

Hardware: Windows PC, Arduino MKR1010 WiFi, Neopixels LED strips, Laser cutter
Software: Arduino IDE, Autodesk Eagle, Adobe Suite, Unreal Engine 4
Development time: 14 weeks

---

Abstract

Horizonte is a clock that adjusts its color gradients and light patterns in real-time according to the device’s location and the current time of day. The clock deploys a WiFi access point for remote configuration and personalization. I wanted to stay away from the feeling of preciseness, I preferred a much less rigid, and non-invasive way of letting you know what part of the day it is. The clock gradually shifts from one stage of the day to the next, taking around 5- 20 minutes to shift depending on the event, location, and date.

The project uses a server-based API hosted on Heroku to acquire the time for each particular ‘sun event’ in the day, such as Dawn, Sunset, etc… The clock only depicts events that happen between Dawn and Dusk. Why? well, these are the first and last stages of sunlight, and if you are a fan of ” From Dusk till Dawn”, those are vampire-safe times.

You can find more about the process, including software and hardware development in this blog post.

Working Prototype

User interface

User task flow

FULL PROCESS

I mainly focused on gradients, however I also tried different approaches, as you can also see below.

I quickly found myself struggling to acquire the control I needed, and after a week or two of trying to make my own gradients based off of Adafruit’s limited examples, I was running out of time to get the other things going. After looking for a while I ran into FastLED, a Neopixel control library developed for Arduino which boasts to be better than the original library. Their examples were very varied, complete, and allowed me to find a couple of useful scripts which I took apart to make my own. Specifically, I heavily relied on the ColorPalette example and the nblendPaletteTowardPalette function found in the docs. The code implemented in the clock goes through nine different gradients, transitioning in and out of them during the day.

The main function takes the following parameters:

• A current time measure in order to determine which sun event should be activated
• The start time for the current day loop. It resets at midnight.
• each of the sun events in millis().

void ChangePalettePeriodically_bottom(unsigned long startMillis,
                                      unsigned long currentMillis,
                                      long interval1,
                                      long interval2,
                                      long interval3,
                                      long interval4,
                                      long interval5,
                                      long interval6,
                                      long interval7,
                                      long interval8,
                                      long interval9
                                     ) {

In terms of wiring and connections, it was pretty straightforward. I used an Arduino MKR 1010 since it had an esp32. From previous research, I knew it had more ram than the Arduino Nano, which was my other option. I hooked a couple of 470-ohm resistances to the input line of each Neopixel Strip. I also added two 470 microfarads capacitors in series. The Adafruit Uberguide for powering Neopixels suggests a 1000 microfarad capacitor, however, I did not have one. All of this was powered by a 5V, 10A power supply. Here’s the schematic below.

Color

In order to represent the horizon, or at least my view of it, I would have to decide on a color palette to use. I started out doing something like the image below, which gave me a general sense of where to go.

The API, the request, the data

After having seen Tom Igoe’s Connected Devices and Networked Interaction class last semester, I felt fairly confident that making my own API was the way to go in order to access the data at any given time from any device and also to control it. Manipulating the data and processing it before sending it to any device was a priority. I decided to make it in NodeJS since I wanted to keep the possibility of incorporating JS into the web client open. I also mounted it on Heroku, which probably isn’t the best option for continuous communication such as WebSockets but works fine for HTTP requests. As I said before, even though my idea was to keep it as real-time as possible the idea of being a couple of seconds or even minutes off track didn’t mean much. The API consists of two routes: /sun and /now. /sun ill return all of the times for each sun event according to UTC. /now will return the current time.

The data is presented as a JSON file. So everytime we make a request, the server writes an updated JSON file which is sent to the Arduino. You can see the JSON formatted below.

To made the request to the server I used the WiFi Web Client example from the Arduino IDE, and later saved it as a JSON using the ArduinoJson library.

This allowed me to access each key-value in the JSON easily, and save it to set the parameters in our Neopixels lights function. Below you can see a glimpse of how it went for saving the data for Dawn. I later expanded this to include an SD card in order to save the dat and access it later in case I did not have an internet connection. Mostly as a fail-proof asset.

int dawnR, dawnG, dawnB;
long dawnMillis;
JSONVar myDataObject;

JSONVar apiSunTimesRequest(JSONVar myDataObject) {
  // assemble the path for the GET message:
  String path = "/sun";

  // send the GET request
  Serial.println("making GET request");
  httpclient.get(path);

  // read the status code and body of the response
  int statusCode = httpclient.responseStatusCode();
  String response = httpclient.responseBody();
  Serial.print("Status code: ");
  Serial.println(statusCode);
  Serial.print("/sun Response: ");
  Serial.println(response);

  //Write the Data to the SD Card
  writeToSD(response);

  // parse the string into a JSONVar object:
  myDataObject = JSON.parse(response);

  dawnMillis = myDataObject["dawn"]["time"]["ms"];
  
  // Set the color for each event
  dawnR = myDataObject["dawn"]["color1"]["r"];
  dawnG = myDataObject["dawn"]["color1"]["g"];
  dawnB = myDataObject["dawn"]["color1"]["b"];
   
    return  myDataObject;
  }
}

apiSunTimesRequest(myDataObject);

Form

I went though several iterations in order to get to a form which satisfied my thirst for a minimal horizon representation. I used blender in order to get a dimensionality and feel for what a wanted, and later used Illustrator in order to create the necessary shapes to later cut the shape with wood.

I quickly discarded the circle idea, and rather went for a more “horizon” kind of look. Such as this one taken by NASA:

When I was satisfied with my 3D model, I decided to makes some illustrator sketches to start building the form . The construction would consist of several wooden layers attached together to achieve the required volume. The ‘Front Side’ , or show side, would have no markings on it, the remaining layers would be the main structure. These would be cut with laser to acquire the best precision possible, and would also allow me to design the frame in order to contain the components out of view.

I also started drawing out possible angles and the necessary depth of the volume, translating into how many layers of wood I would need. I decided on six layers of wood, each with a depth of 6mm. As you can see below, my idea was to make a 45 degree angle in the middle layers in order to place the lights on top

So at first I just played around with the shape of each of the layers on top of each other:

Getting a feel on how I would fit the components inside. As a minimum, the leanest slice would at least have to have 2.6 cms, So that was the measure I used as the scale for the other slices.

After several failed cuts, designs, and a lot of illustrator, I finally arrived at the design below, which divided each of the 5 “out-of-view” pieces to be cut into three separate pieces, each one with a hole for wooden dowels.

Production

The Strips

The strips come in a silicon enclosure, so I cut it off to have more access to the strip itself. The enclosure is so nice I actually felt sad about having to cut it. Besides, it cost a lot.

I then made a little jig in order to be able to keep my lights a distance away while I still worked on them from my computer. See it below:

The structure

First approaches:

Failed design:

After 3 different failed cuts, I arrived at this design. Look at that beauty:

Putting it all together

I pasted the strips on the device using double-sided tape, and also sent the connecting cables inside the frame to the Arduino compartment as you can see below.

Tangible Dynamics (2019)

---

Hardware: Windows PC, Microsoft Kinect 2, Optoma projector, Arduino MKR1010 WiFi, Neopixels LED rings
Software: Touchdesigner, Unity
Programming language: Python, C#
Development time: 5 weeks

---

A playground of luminous spheres and particle animations invited participants play and discover new ways of connecting and creating animations with each other. This project was the final project for my Introduction to Physical Computing class at NYU. It was co-created with Nicole Ginelli, and you can see more of her amazing work here. The idea was the combination of our combined interests in interactive 3D animations, real-time user control, and a desire to put into use several approaches we learned during the semester,

We used a Kinect 2 for depth, luminance, and color tracking. Using this combined approach, we were able to control the tracking in a way where only the spheres would be tracked, and people’s extremities would not.

This video feed was managed and programmed using Touchdesigner, which we used to get X and Y coordinates for each of the tracked objects. We then used the OSC communication protocol to send these values to Unity3D, where we assigned each set of coordinates to control objects inside the particle systems we created with the game engine. Simultaneously, we used SPOUT to send the game engine’s camera view as a texture over to Touchdesigner, where we adjusted colors, contrast, and brightness before video mapping the image on top of the table using a short-throw projector. Finally, we also had an Arduino NANO 33 IOT inside each of the spheres you can observe on the table. Each Arduino controlled an independent neo-pixel LED strip. We used UDP communication to control the color and luminance of each of the neo-pixel strips according to the particle simulation we decided to show. In total, we had four different particle simulations. Each one had with own properties, LED color combination, and audio-reactive composition.

Process

Developer View

Initial Sketches

User Testing

System diagram

Bogotá Cinematheque Inauguration (2019)

---

Hardware: Windows PC, Microsoft Kinect 2, Barco projector
Software: Unity3D, Blender
Development time: 6 weeks

---

Abstract

I was given the honor to develop an interactive audiovisual installation made for the inauguration of Bogotá’s Cinematheque in June 2019, presented by the Art, Science and Technology line of Idartes, CKWeb, and the Cinematheque, as well as the local government. The project is a large-scale projection mapping that allowed users to explore Colombia’s Cinematographic database through the use of their hands. The experience created a virtual world where each movie was represented by a cube alongside some basic information about the film. The users could grab, punch or collide each movie with each other while also enjoying some iconic scenes from Colombia’s rich cinematographic past.

The tracking was made using a Kinect 2 to track the user’s presence and head movement, later to be replicated in the projection. Instead of a parallax effect, the setup was thought in a way the user would feel as if they were reaching out into another reality. The simulation and database control was created in Unity. The software itself presented two states, idle and active. one for user presence and another for user absence. Thanks to everyone who made part of this project, and to the Art, Science and Technology line of Idartes, CKWeb, and the Cinematheque.

Process

MUSA AR (2018)

---

Hardware: IOS and Android devices
Software: Unity3D
Programming language: C#
Development time: 4 weeks

---

MUSA AR is an Augmented reality educational experience made for Bogotá’s Archaeological Museum MUSA in 2018.

The app allowed people to visualize pre-Columbian artifacts including pottery, musical instruments, and jewelry. It was made for a series of educational workshops open to the public around the country. To achieve the maximum detail fidelity, we used photogrammetry to visualize the pieces.

Images were taken by Roberto García Poveda. You can see stills and a video of the app below.

App Use Manual

Radio Parasites (2018)

Hardware: Windows PC, Arduino MKR1010 WiFi, Neopixels LED strips, Laser cutter
Software: Arduino IDE, Autodesk Eagle, Adobe Suite, Unreal Engine 4
Programming language: Arduino C/C++
Development time: 14 weeks

Abstract

Radio Parasites is an audiovisual performance which utilizes recorded radio signals to create generative sound and control images which reflect about the intertwinement between cosmic radio waves, geological movements, and our place among these phenomenon. The piece is a reflection about an electromagnetic ecology and how we and the world we inhabit are bombarded by these radio frequencies. It is a data sonification of invisible forces that holds information from geological ground stations to satellites.

Developed with Atractor Collective in 2018, presented at the Museum of Contemporary Art in Panama.

The visuals were controlled with an xbox controller and through OSC with Touchdesigner. Used several GLSL video filters on top of content generated with Unity3D.

User Interface