Video for py5 #266

villares · 2023-03-23T20:16:42Z

villares
Mar 23, 2023
Collaborator

Hey folks,

Suppose I want to display some video in the background of a sketch... what do you suggest as a replacement of Processing Foundation's Video library?

Answered by villares

Mar 27, 2023

Maybe we should avoid Processing Video and use OpenCV instead... this works, adapted from my webcam capture code, with a few tweaks:


import py5
import cv2
import numpy as np
from py5 import create_image_from_numpy

movie = cv2.VideoCapture('launch2.mp4')
movie_width = int(movie.get(cv2.CAP_PROP_FRAME_WIDTH))
movie_height = int(movie.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = movie.get(cv2.CAP_PROP_FPS)

py5_img = None

def setup():
    py5.size(movie_width, movie_height)
    py5.frame_rate(fps)

def draw():
    global py5_img
    success, frame = movie.read()  # frame is a numpy array
    if success:
#         gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
#         edges = cv2.Canny(gray, 10…

View full answer

hx2A · 2023-03-24T00:10:08Z

hx2A
Mar 24, 2023
Maintainer

First, why replace the Processing Foundation's Video library? It works just fine with py5. You'll need to add the video jar to your classpath, which can be done by creating a "jars" subdirectory and putting it in there. Note that py5 supports a user-defined movie_event() function that gets called for each new frame, if this is how you want to use the library. I believe the Movie object is a subclass of PImage, so you can pass it as a param to background().

If you don't want to use Processing's Video library, you can also use opencv. Everything you know how to do with webcams would be applicable here.

4 replies

villares Mar 24, 2023
Collaborator Author

Hmm, you have great points. I guess it still feels a bit awkward for me to move jars around, and, as I got burned by Processing Foundation's libraries not working on different platforms in Processing Python mode anymore (both Serial and Video had problems on Windows), so my gut instinct told me I should move away from them, but this is just prejudice, I should definitely try the Processing Video library!

hx2A Mar 25, 2023
Maintainer

(both Serial and Video had problems on Windows)

If the Video library had problems on Windows, I doubt py5 will solve them. What were the problems? How does the Video library work for Processing in the PDE? I imagine the tool the Video library uses to decode videos, gstreamer, might have some install complications on Windows, but I would be surprised if it couldn't work at all.

villares Mar 25, 2023
Collaborator Author

It had problems only on Python mode... I'll see if I can make it work with py5... but it will take some time.

UPDATE: Today I could confirm video is working on Java but not on Python Mode on Windows:

UnsatisfiedLinkError: C:\Users\alexandre.villares\Documents\Processing\libraries\video\library\windows64\avcodec-58.dll: Can't find dependent libraries
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
	at java.lang.Runtime.load0(Runtime.java:809)
	at java.lang.System.load(System.java:1086)
	at jycessing.LibraryImporter.recursivelyLoadNativeLibraries(Unknown Source)
	at jycessing.LibraryImporter.addLibrary(Unknown Source)
	at jycessing.LibraryImporter$1.__call__(Unknown Source)
	at org.python.core.PyObject.__call__(PyObject.java:480)
	at org.python.core.PyObject.__call__(PyObject.java:484)
	at org.python.pycode._pyx11.f$0(sketch_230325c_video.pyde:1)
	at org.python.pycode._pyx11.call_function(sketch_230325c_video.pyde)
	at org.python.core.PyTableCode.call(PyTableCode.java:171)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.Py.runCode(Py.java:1614)
	at org.python.core.Py.exec(Py.java:1658)
	at org.python.pycode._pyx10.f$0(C:/Users/alexandre.villares/AppData/Local/Temp/sketch_230325c_video3342077887742644558/sketch_230325c_video.pyde:96)
	at org.python.pycode._pyx10.call_function(C:/Users/alexandre.villares/AppData/Local/Temp/sketch_230325c_video3342077887742644558/sketch_230325c_video.pyde)
	at org.python.core.PyTableCode.call(PyTableCode.java:171)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.Py.runCode(Py.java:1614)
	at org.python.core.Py.exec(Py.java:1658)
	at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:276)
	at jycessing.PAppletJythonDriver.processSketch(Unknown Source)
	at jycessing.PAppletJythonDriver.findSketchMethods(Unknown Source)
	at jycessing.Runner.runSketchBlocking(Unknown Source)
	at jycessing.mode.run.SketchRunner.lambda$startSketch$3(Unknown Source)
	at java.lang.Thread.run(Thread.java:748)

Next step is to try with py5... I have copied all the necessary files, tomorrow at Sesc I'll try and test it...

hx2A Mar 25, 2023
Maintainer

It had problems only on Python mode... I'll see if I can make it work with py5... but it will take some time.

Sounds good! Keep me in the loop as you make progress.

villares · 2023-03-26T20:36:04Z

villares
Mar 26, 2023
Collaborator Author

After downloading and installing the Processing Foundation's Video library from the Processing 3.5.4 IDE, I copied the contents of the <user>\Documents\Processing\libraries\video\library to a jars folder next to my py5 sketch, my_video.py. The video sample I'm using launch2.mp4 is on the same folder as my_video.py as it is expected for py5 (in Processing Java & Python Mode you need to put it inside a data folder).

The Video library for Procesing 3.54 (based on GStreamer 1.16.2) kind of worked, freezing after some time. The newest Video for Processing 4.X is a bit different and I couldn't make it work at all.

import py5
from processing.video import *

def setup():
    global movie
    py5.size(560, 406)
    py5.background(0)
    # Load and play the video in a loop
    this = py5.get_current_sketch()
    movie = Movie(this, 'launch2.mp4')
    movie.loop()

def draw():
    if movie.available():
        movie.read() 
    py5.image(movie, 0, 0, py5.width, py5.height)

py5.run_sketch()

Defining a movieEvent function didn't work for me.

Using the Video library for Processing 4.2 I got this:

We could not find a system-wide or bundled installation of GStreamer, but video might still work if GStreamer was placed somewhere else
Seems like you are trying to use GStreamer native libraries older than 1.20, which are not supported.
py5 encountered an error in your code:File "PSurfaceNone.java", line 356, in processing.core.PSurfaceNone$AnimationThread.run

File "PSurfaceAWT.java", line 1386, in processing.awt.PSurfaceAWT$9.callDraw
File "PApplet.java", line 2088, in processing.core.PApplet.handleDraw
File "Sketch.java", line 196, in py5.core.Sketch.setup
File "jdk.proxy2.$Proxy8.java", line -1, in jdk.proxy2.$Proxy8.run_method
File "org.jpype.proxy.JPypeProxy.java", line -1, in org.jpype.proxy.JPypeProxy.invoke
File "org.jpype.proxy.JPypeProxy.java", line -2, in org.jpype.proxy.JPypeProxy.hostInvoke
File "processing.video.Movie.java", line -1, in processing.video.Movie.<init>
File "processing.video.Movie.java", line -1, in processing.video.Movie.initGStreamer
File "processing.video.Video.java", line -1, in processing.video.Video.init
File "processing.video.Video.java", line -1, in processing.video.Video.initImpl
File "Gst.java", line 453, in org.freedesktop.gstreamer.Gst.init
File "Gst.java", line 473, in org.freedesktop.gstreamer.Gst.init
File "Gst.java", line 144, in org.freedesktop.gstreamer.Gst.getVersion
File "GstAPI.java", line 37, in org.freedesktop.gstreamer.lowlevel.GstAPI.<clinit>
File "GstNative.java", line 46, in org.freedesktop.gstreamer.lowlevel.GstNative.load
File "GstNative.java", line 56, in org.freedesktop.gstreamer.lowlevel.GstNative.load

Exception: Java Exception
The above exception was the direct cause of the following exception:
File "C:\Users\alexandre.villares\Documents\video\my_video.py", line 22, in _py5_faux_setup
    16   def setup():
 (...)
    18       py5.size(560, 406)
    19       py5.background(0)
    20       # Load and play the video in a loop
    21       this = py5.get_current_sketch()
--> 22       movie = Movie(this, 'launch2.mp4')
    23       movie.loop()
    ..................................................
     Movie = <java class 'processing.video.Movie'>
     this = <py5.sketch.Sketch object at 0x0000021DBA421370>
    ..................................................

java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError: Could not load library: gstreamer
>>>

3 replies

hx2A Mar 27, 2023
Maintainer

I am on my phone right now and have a few questions. If you are on windows, can you do this in an anaconda environment and install gstreamer through that? I have only used the video library on Linux. I know that library changed slightly from 3 to 4 (one change was a PR from me). The movie event function is I believe movie_event() (it is mentioned on the Sketch class page in the reference docs). If that doesn't get called, it might be a bug.

villares Mar 27, 2023
Collaborator Author

Curious that you mention it! Now I remember I tried def movie_event(m): out of intuition and was surprised that m was not the movie object I expected with a .read() method (it seems it was a Py5Image object!).

Microsoft managed to wreck my dual boot Windows, somehow it broke without me logging in for months, so I'll only have access to a computer with Windows next Tuesday at Sesc (the community center closes Mondays).

I'll see if I can set a new env with anacaonda then.

villares Mar 27, 2023
Collaborator Author

No luck on Linux either :(

can't load library gstrtspserver-1.0 (gstrtspserver-1.0|libgstrtspserver-1.0|libgstrtspserver-1.0-0) with -Djna.library.path=/usr/lib64. Last error:java.lang.UnsatisfiedLinkError: Unable to load library 'gstrtspserver-1.0':
libgstrtspserver-1.0.so: não é possível abrir arquivo compartilhado: Arquivo ou diretório inexistente
libgstrtspserver-1.0.so: não é possível abrir arquivo compartilhado: Arquivo ou diretório inexistente

can't load library gstvalidate-1.0 (gstvalidate-1.0|libgstvalidate-1.0|libgstvalidate-1.0-0) with -Djna.library.path=/usr/lib64. Last error:java.lang.UnsatisfiedLinkError: Unable to load library 'gstvalidate-1.0':
libgstvalidate-1.0.so: não é possível abrir arquivo compartilhado: Arquivo ou diretório inexistente
libgstvalidate-1.0.so: não é possível abrir arquivo compartilhado: Arquivo ou diretório inexistente

can't load library gstvalidate-default-overrides-1.0 (gstvalidate-default-overrides-1.0|libgstvalidate-default-overrides-1.0|libgstvalidate-default-overrides-1.0-0) with -Djna.library.path=/usr/lib64. Last error:java.lang.UnsatisfiedLinkError: Unable to load library 'gstvalidate-default-overrides-1.0':
libgstvalidate-default-overrides-1.0.so: não é possível abrir arquivo compartilhado: Arquivo ou diretório inexistente
libgstvalidate-default-overrides-1.0.so: não é possível abrir arquivo compartilhado: Arquivo ou diretório inexistente

can't load library avresample (avresample|libavresample|libavresample-0) with -Djna.library.path=/usr/lib64. Last error:java.lang.UnsatisfiedLinkError: Unable to load library 'avresample':
libavresample.so: não é possível abrir arquivo compartilhado: Arquivo ou diretório inexistente
libavresample.so: não é possível abrir arquivo compartilhado: Arquivo ou diretório inexistente

can't load library fdk_aac (fdk_aac|libfdk_aac|libfdk_aac-0) with -Djna.library.path=/usr/lib64. Last error:java.lang.UnsatisfiedLinkError: Unable to load library 'fdk_aac':
libfdk_aac.so: não é possível abrir arquivo compartilhado: Arquivo ou diretório inexistente
libfdk_aac.so: não é possível abrir arquivo compartilhado: Arquivo ou diretório inexistente

can't load library openh264 (openh264|libopenh264|libopenh264-0) with -Djna.library.path=/usr/lib64. Last error:java.lang.UnsatisfiedLinkError: Unable to load library 'openh264':
libopenh264.so: não é possível abrir arquivo compartilhado: Arquivo ou diretório inexistente
libopenh264.so: não é possível abrir arquivo compartilhado: Arquivo ou diretório inexistente

villares · 2023-03-27T23:03:54Z

villares
Mar 27, 2023
Collaborator Author

Maybe we should avoid Processing Video and use OpenCV instead... this works, adapted from my webcam capture code, with a few tweaks:


import py5
import cv2
import numpy as np
from py5 import create_image_from_numpy

movie = cv2.VideoCapture('launch2.mp4')
movie_width = int(movie.get(cv2.CAP_PROP_FRAME_WIDTH))
movie_height = int(movie.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = movie.get(cv2.CAP_PROP_FPS)

py5_img = None

def setup():
    py5.size(movie_width, movie_height)
    py5.frame_rate(fps)

def draw():
    global py5_img
    success, frame = movie.read()  # frame is a numpy array
    if success:
#         gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
#         edges = cv2.Canny(gray, 100, 80) 
#         edges_rgb_npa = cv2.cvtColor(edges, cv2.COLOR_GRAY2BGR)
#         blended_rgb_npa = cv2.addWeighted(frame, 0.6, edges_rgb_npa, 0.4, 0)
#         py5_img = create_image_from_numpy(blended_rgb_npa, 'RGB', dst=py5_img)
        py5_img = create_image_from_numpy(frame, 'RGB', dst=py5_img)
        # display image
        py5.image(py5_img, 0, 0)
    else:  # If it can't read frame, try starting again
        movie.set(cv2.CAP_PROP_POS_FRAMES, 0)

    if py5.frame_count % 30 == 0:
        py5.window_title(f'FR: {py5.get_frame_rate():.1f}')

def exiting():
    print('over and out')
    movie.release()

py5.run_sketch()

More ideas at: https://vuamitom.github.io/2019/12/13/fast-iterate-through-video-frames.html

3 replies

hx2A Mar 30, 2023
Maintainer

Does Processing Video work at all on Windows when run through the PDE? I'd be surprised if it did not. However, we want to provide folks with a way that works on all systems and does not cause headaches to get set up. OpenCV seems to be that solution.

villares Mar 30, 2023
Collaborator Author

This is what I got last week using the Processing IDE:

[...] Today I could confirm video is working on Java but not on Python Mode on Windows: [...]

Yeah... I'd like to collect some tips on how to do things I used to do with Processing Video, like changing the movie speed...
As you can see on this example I could kind of recreate a looping video...

Someone (probably not me or you) could create a really nice "py5 library" nicely encapsulating OpenCV video features :)

hx2A Mar 31, 2023
Maintainer

Yeah... I'd like to collect some tips on how to do things I used to do with Processing Video, like changing the movie speed...
As you can see on this example I could kind of recreate a looping video...

Is there a bit of a stutter when you loop around to the beginning? It's a hard thing to get rid of.

Someone (probably not me or you) could create a really nice "py5 library" nicely encapsulating OpenCV video features :)

Yes, or to start, write some documentation for this!

Uh oh!

Video for py5 #266

Uh oh!

villares Mar 23, 2023 Collaborator

Replies: 3 comments · 10 replies

Uh oh!

hx2A Mar 24, 2023 Maintainer

Uh oh!

villares Mar 24, 2023 Collaborator Author

Uh oh!

hx2A Mar 25, 2023 Maintainer

Uh oh!

Uh oh!

villares Mar 25, 2023 Collaborator Author

Uh oh!

hx2A Mar 25, 2023 Maintainer

Uh oh!

Uh oh!

villares Mar 26, 2023 Collaborator Author

Uh oh!

hx2A Mar 27, 2023 Maintainer

Uh oh!

villares Mar 27, 2023 Collaborator Author

Uh oh!

villares Mar 27, 2023 Collaborator Author

Uh oh!

Uh oh!

villares Mar 27, 2023 Collaborator Author

Uh oh!

hx2A Mar 30, 2023 Maintainer

Uh oh!

villares Mar 30, 2023 Collaborator Author

Uh oh!

hx2A Mar 31, 2023 Maintainer

villares
Mar 23, 2023
Collaborator

Replies: 3 comments 10 replies

hx2A
Mar 24, 2023
Maintainer

villares Mar 24, 2023
Collaborator Author

hx2A Mar 25, 2023
Maintainer

villares Mar 25, 2023
Collaborator Author

hx2A Mar 25, 2023
Maintainer

villares
Mar 26, 2023
Collaborator Author

hx2A Mar 27, 2023
Maintainer

villares Mar 27, 2023
Collaborator Author

villares Mar 27, 2023
Collaborator Author

villares
Mar 27, 2023
Collaborator Author

hx2A Mar 30, 2023
Maintainer

villares Mar 30, 2023
Collaborator Author

hx2A Mar 31, 2023
Maintainer