Associating videos to characters generatively

edited January 2018 in Library Questions

Hi

I'm trying to create a movie-based typeface. I want each letter to create a movie object. for example letter X needs to call X.mov and become an item in and array.

   currentLetter = new Movie[str.length()];
  for (int i=0; i<str.length(); i++) {
    currentLetter[i] = new Movie(this, str.charAt(i)+".mov");
    currentLetter[i].jump(random(currentLetter[i].duration()));
  }

this returns:

"Could not load movie file .mov"

here is the entire code:

String str = "abab aba";
int wordsNum = 1;

int t = 0;

int[] wordsLength;

float kerning = -130;
float leading = 0;
float charLocation = 0;
float lineLocation = 0;

float fontHeight = 120;
float fontWidth = 160;

float horMargin = 100;
float vertMargin = 100;

Movie[] currentLetter;


import processing.video.*;
void setup() {
  size(1200, 900);
  frameRate(30);
  // finding out how many words are there    (important for the wordsLength array)

  for (int i = 0; i<str.length(); i++) {
    if (str.charAt(i)==' ') {
      wordsNum++;
    }
  }
  println("number of words is: ", wordsNum);


  int letCount = 0;

  wordsLength = new int[wordsNum];
  for (int i = 0; i<str.length(); i++) {
    if (str.charAt(i)==' ') {
      wordsLength[t]=letCount;
      letCount=0;
      t++;
    } else {
      letCount++;
    }
  }
  println(wordsLength);

  currentLetter = new Movie[str.length()];
  for (int i=0; i<str.length(); i++) {
    currentLetter[i] = new Movie(this, str.charAt(i)+".mov");
    currentLetter[i].jump(random(currentLetter[i].duration()));
  }


  t=0;
}

void draw() {
  background(255);
  blendMode(DARKEST);
  charLocation = 0;
  lineLocation = 0;
  t = 0;
  for (int let = 0; let < str.length(); let++) {
    float letterxLocation = width-horMargin+charLocation-fontWidth;
    image(currentLetter[let], letterxLocation, vertMargin+lineLocation, fontWidth, fontHeight);

    // Character Location

    charLocation = charLocation - fontWidth - kerning;

    // checking if it's a new word

    if (str.charAt(let)==' ' && t < wordsNum) {

      // creatng a new line

      if (width-horMargin+charLocation-wordsLength[t]*(fontWidth+kerning) < horMargin) {
        charLocation = 0;
        lineLocation = lineLocation + fontHeight + leading;
      }
      t++;
    }
  }
}

void movieEvent(Movie m) {
  m.read();
}

Any ideas? :) Thank you!

Answers

  • I don't think you want to create multiple movie files playing the same movie. Instead, you could play a movie file (I mean, call your movie.play() function) and then whenever you want to place your movie frame, you just provide the coordinates via image(), exactly as you have done in your previous post:

    https://forum.processing.org/two/discussion/25854/movie-files-do-not-show-no-error#latest

    Kf

  • Hi

    Thank you for your answer I want the movies to play independently at from different starting points. that's why I'm making separate movie files.

  • I see it causes a lot of performance issues. Is there possibly a more efficient way?

  • edited January 2018

    copied from another thread

    jeremydouglass 9:01AM

    What is the video content? You should probably use sprites for this, not movies. I would expect a major performance difference.

    You can also optimize lookups by doing only 26 (or however many characters) passes. So, load the current sprite sheet state for the a animation, then draw every a on the screen (they are all synchronized, right?) then b, then c... now no matter how many characters you type, you are only doing 26 lookups. This is particularly easy if you use a fixed-width "font".

  • Thank you for you answer :) The animations are supposed to be **not **synchronized.

    I'm still contemplating the fixed width. I'd be happy to make it not-fixed in the future.

    I will try to use sprites asap.

  • Hope it is helpful! Let us know here how it goes with sprites.

    Please do not post near-duplicates: https://forum.processing.org/two/discussion/25876/frame-rate-multiple-videos-and-recording

    If you are working on several closely related issues on a single aspect of a single piece of code (like typing multiple videos, and performance), then post updated code as a new message on your existing thread. That way all the information on what you are doing is in one place, and it is easier for us to help you. Duplicates are hard work for most of the forum moderators and volunteers, and it makes them unhappy.

  • (duplicate thread deleted)

  • @jeremydouglass, @koogs ok thank you!

    FYI I've got a huge improvement in performance once added "P2D" to size value. Still will try the sprites angle.

  • edited January 2018

    Great!

    While premature optimization is often unwarranted, optimizing for letter differences when they are resource-intensive is an interesting problem.

    Note that even with non-synchronized movies, if the clips are short then you can constrain then to a few time alignments -- e.g. 3-second clips whose starts are aligned to 0,1,2 seconds. Now instead of up to 26 (or whatever) lookups, you have a maximum of up to three times that many -- 78 -- but that still isn't as bad as doing an individual sprite lookup for each of 2000 different character-frames. In principle: set a maximum complexity which sets a maximum execution load.

    Due to the nature of letter frequency distribution in language, typing in English will mean that over two-thirds of the characters on the screen will be only nine letters -- ETOINSHRD.

    https://en.wikipedia.org/wiki/Letter_frequency#Relative_frequencies_of_letters_in_the_English_language

    If you want optimize while keeping the visual complexity of the work high, give letters like ETOIN more alignment points as they will be more visible "synchronized" -- but not too many, as they represent the majority of your load.

    You can also do this dynamically, by alignment points as new letters are typed. So as you type the letter "e" in a string, each letter e starts aligned with (for example, using any series):

    1,2,1,2, 1,2,3,1,2,3, 1,2,3,4,1,2,3,4, 1,2,3,4,5,1,2,3,4,5...
    

    This means that two letters next two each other will never have the same alignment. A 1000 character paragraph will contain on average 120 Es; they will be covered by 11 different sprite images on any given frame.

    Or you could use the known frequency of the word ahead of time and assign random 10 options to E, 5 to S, 2 to Z.

    Of course, if you want the video font to feel like language, you may actually want tighter alignments -- only 1-2 at most per letter, possibly only one. Then the pile of flickering videos will look like a Zipf's distribution as they change together in different ratios -- that's how a page of natural language looks, with many shapes in different places all the same.

  • image(currentLetter[let], letterxLocation, vertMargin+lineLocation, fontWidth, fontHeight)
    

    there's also a resize of each video going on. resizing things in draw() is bad.

    ETOINSHRD.

    https://en.wikipedia.org/wiki/Etaoin_shrdlu

  • edited January 2018

    good point. resizing videos inside draw on every frame can be really bad -- depending, it could have more negative impact on performance than every other positive optimization you could make all put together.

  • edited January 2018

    So I've tried using sprites and seems like it is not behaving better than using movie files. strangely enough, performence using P2D was much worse (attaching the code at the bottom).

    a few questions:

    (1) if I enter the original height an width of the video to the size value does it still act as if it is resizing it?

    (2) Jeremy please tell me if I understand your answer correctly - will using a limited number of "starting points" improve performance?

    (3) What if I will change a starting point only for each instance of the letter. let's say we use the word "Massachusetts", the starting points will be 1112211131123 That way, same letter won't have a similar starting point, but new starting points will be limited only to repeating letters. the question here is, should that improve performance?

    Thank you for all your help you guys are wonderful :)

    String str = "אשגךלגךלגךלגךלגךלגךלגךלגךלגךלגךלבל";
    
    // typographic elements
    
    float hor_margin = 30;
    float vert_margin = 30;
    float kerning = 50;
    float leading = 100;
    
    int[] letter_frame = new int[str.length()];;
    
    String  zeros = "000";
    PImage sprite;
    
    void setup(){
      size(800, 600);
    
      for(int i = 0; i <str.length(); i++){
      letter_frame[i]=int(random(0,300));
    
      }
    }
    
    // increments the letter
    
    void letter(char which_letter, float x_loc, int i){  
        letter_frame[i]++;
        if(letter_frame[i]<10){
         zeros="000"; 
        }
        if (letter_frame[i]  >= 10){
         zeros ="00"; 
        }
        if (letter_frame[i] >= 100){
         zeros = "0"; 
        }
        if (letter_frame[i] >= 1000){
         zeros = ""; 
        }
        if (letter_frame[i] == 3617){
        zeros = "000";
        letter_frame[i] = 1;
        }
        sprite =loadImage(which_letter+zeros+letter_frame[i]+".png");
        image(sprite, x_loc, 0);
    
        blendMode(MULTIPLY);
    
    }
    
    void draw(){
      background(255);
    
      // goes through each letter and draws it
    
      for (int i = 0; i < str.length(); i++){
       letter(str.charAt(i), width-i*kerning, i);
     }
    
             fill(0);
      textSize(16);
      text("Frame rate: " + int(frameRate), 10, 20);
    
    }
    
  • sprite =loadImage(which_letter+zeros+letter_frame[i]+".png");

    don't loadImages in draw() (or anything called from draw()). do it in setup() and reference it.

  • are you sure? this will load 3617*26=94042 images at the same time

  • edited January 2018

    @dehyde===

    • your code for videos cannot run: in setup() your array for movies contains null movies (called .mov) for each ' ' char: that's why you get the error you posted. That is easy to solve.

    • in draw() && in your first loop() you use str.length: that is the same error; you have to use your array length (after removing the null movies)

    • i have tested changing fontWidth && fontHeight to the real videos width && height in order to get rid of resizing: with 5 words and 4 videos(160/140) it works at 28fps

    • Not tested with 26 movies!

  • are you sure? this will load 3617*26=94042 images at the same time

    and loading 94042 images n at a time in draw will be SLOW. you need to compromise somewhere.

  • you forget that we haven't seen your assets, we don't know how big they are or how many frames they contain.

  • edited January 2018

    I've removed the currentLetter[i].jump and intrestingly enough the behaviour is the same and letter still seem randomizes.

    @koogs

    each asset is 120 fps, and contains ~3600 frames on 122*160 pixels. I don't mind long loading time, I was wondering though if the memory can hold that much data without burdening the system.

    @akenaton

    Currently I have a code that functions rather well (will attach to bottom), 20 fps for around 250 characters.

    It's showing a lot of errors though and gets stuck occasionally.

    java.lang.NullPointerException
        at processing.opengl.Texture.copyBufferFromSource(Texture.java:827)
        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at processing.video.Movie.read(Unknown Source)
        at draft_2_typoterry.movieEvent(draft_2_typoterry.java:129)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at processing.video.Movie.fireMovieEvent(Unknown Source)
        at processing.video.Movie.invokeEvent(Unknown Source)
        at processing.video.Movie$1.bufferFrame(Unknown Source)
        at org.gstreamer.elements.BufferDataAppSink$AppSinkNewBufferListener.newBuffer(BufferDataAppSink.java:163)
        at org.gstreamer.elements.AppSink$2.callback(AppSink.java:184)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.jna.CallbackReference$DefaultCallbackProxy.invokeCallback(CallbackReference.java:485)
        at com.sun.jna.CallbackReference$DefaultCallbackProxy.callback(CallbackReference.java:515)
    

    and this:

    java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at processing.video.Movie.post(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at processing.core.PApplet$RegisteredMethods.handle(PApplet.java:1427)
        at processing.core.PApplet$RegisteredMethods.handle(PApplet.java:1420)
        at processing.core.PApplet.handleMethods(PApplet.java:1614)
        at processing.core.PApplet.handleDraw(PApplet.java:2467)
        at processing.opengl.PSurfaceJOGL$DrawListener.display(PSurfaceJOGL.java:859)
        at jogamp.opengl.GLDrawableHelper.displayImpl(GLDrawableHelper.java:692)
        at jogamp.opengl.GLDrawableHelper.display(GLDrawableHelper.java:674)
        at jogamp.opengl.GLAutoDrawableBase$2.run(GLAutoDrawableBase.java:443)
        at jogamp.opengl.GLDrawableHelper.invokeGLImpl(GLDrawableHelper.java:1293)
        at jogamp.opengl.GLDrawableHelper.invokeGL(GLDrawableHelper.java:1147)
        at com.jogamp.newt.opengl.GLWindow.display(GLWindow.java:759)
        at com.jogamp.opengl.util.AWTAnimatorImpl.display(AWTAnimatorImpl.java:81)
        at com.jogamp.opengl.util.AnimatorBase.display(AnimatorBase.java:452)
        at com.jogamp.opengl.util.FPSAnimator$MainTask.run(FPSAnimator.java:178)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
    Caused by: java.lang.NullPointerException
        at java.util.LinkedList.unlink(LinkedList.java:211)
        at java.util.LinkedList.remove(LinkedList.java:526)
        at processing.opengl.Texture.disposeSourceBuffer(Texture.java:841)
        ... 23 more
    

    code:

    String str = "פנומנולוגיה הוא זרם או מתודה בפילוסופיה של סופ המאה התשע עשרה ותחילת המאה העשרים זוהי פילוסופיה תיאורית של הניסיון שמטרתה היא תיאור התהליך שבו התנסות מסוימת מגיעה להכרתנו באופן ישיר ללא צורך להיעזר בתאוריות דעות קדומות מסקנות או השערות מתחומים אחרים";
    int wordsNum = 2;
    
    
    int t = 0;
    
    int[] wordsLength;
    
    float fontMultiplier = 2;
    
    float kerning = -142*fontMultiplier;
    float leading =-200;
    float charLocation = 0;
    float lineLocation = 0;
    
    float fontHeight = 122*fontMultiplier;
    float fontWidth = 160*fontMultiplier;
    
    float horMargin = 100;
    float vertMargin = 100;
    
    Movie[] currentLetter;
    
    
    import processing.video.*;
    void setup() {
      size(1200, 900, P2D);
    
      //frameRate(2);
      // finding out how many words are there    (important for the wordsLength array)
    
      for (int i = 0; i<str.length(); i++) {
        if (str.charAt(i)==' ') {
          wordsNum++;
        }
      }
      println("number of words is: ", wordsNum);
    
    
      int letCount = 0;
    
      wordsLength = new int[wordsNum];
      for (int i = 0; i<str.length(); i++) {
        if (str.charAt(i)==' '||i==str.length()-1) {
          wordsLength[t]=letCount;
          letCount=0;
          t++;
        } else {
          letCount++;
        }
      }
      println(wordsLength);
    
      currentLetter = new Movie[str.length()];
      for (int i=0; i<str.length(); i++) {
        if(str.charAt(i)!=' '){
        currentLetter[i] = new Movie(this, str.charAt(i)+".mov");
        //currentLetter[i].frameRate(2);
        currentLetter[i].loop();
        //currentLetter[i].jump(random(0,3));
        }
        else{
        currentLetter[i] = new Movie(this, "space.mov");
        }
    
      }
    
    
      t=0;
    }
    
    void draw() {
      background(255);
      blendMode(MULTIPLY);
      charLocation = 0;
      lineLocation = 0;
      t = 0;
      for (int let = 0; let < str.length(); let++) {
        float letterxLocation = width-horMargin+charLocation-fontWidth;
        image(currentLetter[let], letterxLocation, vertMargin+lineLocation);
    
    
    
        // Character Location
    
        charLocation = charLocation - fontWidth - kerning;
    
        // checking if it's a new word
    
        if (str.charAt(let)==' ' && t < wordsNum) {
    
          // creatng a new line
    
          if (width-horMargin+charLocation-wordsLength[t+1]*(fontWidth+kerning) < horMargin) {
            charLocation = 0;
            lineLocation = lineLocation + fontHeight + leading;
          }
          t++;
        }
    
    
    
      }
            fill(0);
      textSize(16);
      text("Frame rate: " + int(frameRate), 10, 20);
    //  saveFrame("output/framer_####.jpg");
    }
    
    void movieEvent(Movie m) {
      m.read();
    }
    
Sign In or Register to comment.