FFmpeg’s fps filter, documented

Posted by on 30 Apr 2020 | Tagged as: FFmpeg, robobait, software engineering, web technology

The FFmpeg media editing software is a valuable tool, but its documentation is only barely adequate. It certainly does not answer all the questions I have, as a user trying to understand why FFmpeg is not doing what I want it to do.

Fortunately, FFmpeg is open source, so when the documentation fails, one can read the source. I wanted to learn about presentation time stamps and time bases. The fps video filter source code, in file libavfilter/vf_fps.c, was an instructive read.

I took what I learned from reading that source, and did a complete rewrite of the fps filter documentation. It is longer than the original fps filter documentation (as archived in April 2020 — you can check if the present documentation is any better). I believe the rewrite is more complete and more accurate. I contributed the rewrite to the FFmpeg project. I submitted it as a patch to the ffmpeg developers list. Discussion continues. I don’t know if this contribution will ultimately get accepted.

So, for the benefit of FFmpeg users who are web-searching for answers, here is my documentation of FFmpeg’s fps video filter.


fps

Make a new video from the frames and presentation time stamps (PTSs) of the input. The new video has a specified constant frame rate, and new PTSs. It generally keeps frames from the old video, but might repeat or drop some frames. You can choose the method for rounding from input PTS to output PTS. This affects which frames fps keeps, repeats, or drops.

It accepts the following parameters:

fps

The output frame rate, in frames per second. May be an integer, real, or rational number, or an abbreviation. The default is 25.

start_time

A time, in seconds from the start of the input stream, which fps converts to an input starting PTS and an output starting PTS. If set, fps drops input frames which have PTSs less than the input starting PTS. If not set, the input and output starting PTSs are zero, but fps drops no input frames based on PTS. (See details below.)

round

Rounding method to use when calculating output PTSs from input PTSs. If the calculated output PTS is not exactly an integer, then the value determines which neighbouring integer value fps selects.

Possible values are:

  • zero
    • round towards 0
  • inf
    • round away from 0
  • down
    • round towards -infinity
  • up
    • round towards +infinity
  • near
    • round to nearest (midpoints round away from 0)

The default is near.

eof_action

Action which fps takes with the final input frame. The input video passes in a final input PTS, which fps converts to an output PTS limit. fps drops any input frames with a PTS at or after this limit.

Possible values are:

  • round
    • Use same rounding method as for other frames.
  • pass
    • Round the ending input PTS using up. This might make fps include one last input frame. 

The default is round.

Alternatively, the options may be specified as a flat string:  fps[:start_time[:round]].

fps makes an output video with consecutive integer PTSs, and with a time base set to the inverse of the given frame rate. fps keeps, repeats, or drops input frames, in sequence, to the output video. It does so according to their input PTSs, as converted to seconds (via the input time base), then rounded to output PTSs. 

fps sets output PTSs in terms of a timeline which starts at zero. For any output frame, the integer PTS multiplied by the time base gives a value in seconds on that timeline. If the start_time parameter is not set, or is zero, the first output frame’s PTS is zero. Otherwise, the first PTS is the output starting PTS calculated from the start_time parameter. 

fps interprets input PTSs in terms of the same timeline. It multiplies each input frame’s PTS by the input time base, to get a value in seconds on the timeline. It rounds that value to an integer output PTS. For example, if the input video has a frame rate of 30 fps, a time base of 1/30 seconds, and a first frame with a PTS of 300, then fps treats that first frame as occurring 10 seconds (= 300 * 1/30) after the start of the video, even though it is the first frame.

Setting a start_time value allows for padding/trimming at the start of the input. For example, you can set start_time to 0, to pad the beginning with repeats of the first frame if a video stream starts after the audio stream, or to trim any frames with a negative PTS. When start_time is not set, the fps filter does not pad or trim starting frames, as long as they contain PTSs.

See also the setpts and settb filters.

Details

fps outputs exactly one frame for each output PTS. If there is exactly one input frame with an input PTS which converts to the current output PTS, fps keeps (outputs) that frame. If there are multiple frames which convert to the same output PTS, fps outputs the final frame of that group, and drops the previous frames. 

If the input frame PTS converts to an output PTS later than the current output PTS, fps repeats the previously output frame as the current frame. When this happens for the first input frame,  fps “pads” — outputs repetitions of — that first frame until the output PTS reaches the value converted from that first frame’s input PTS. 

fps always drops input frames which have no PTS set, regardless of the start_time parameter. 

The frame rate value must be zero or greater. It may be provided in a variety of forms. Each form is converted into a rational number, with an integer numerator and denominator. 

  • An integer number, e.g. 25. This converts to the rational number 25/1.
  • A real number, e.g. 3.14145926. This converts to a rational number, e.g. 954708/303893
  • A rational number. The numerator and denominator may be either integers or real numbers. e.g. 30/1.001 or -30000/-1001, which both convert to 30000/1001. The denominator must be non-zero.
  • An abbreviation. e.g ntsc as 30000/1001ntsc-film as 24000/1001. See the complete list at the “Video rate” section in the ffmpeg-utils(1) manual.

fps defines a sync point on the timeline, where one input PTS and one output PTS occur at the same moment. It calculates other PTSs as time offsets from this sync point. This affects the details of rounding. If start_time is set, then fps uses it to calculate input and output PTSs, and makes them the sync point. Otherwise, input and output PTS of zero are the sync point.

Note that fps does not assume that input frames are separated by exactly 1/frame_rate seconds. It takes the input PTSs literally. If the increment of PTS between frames varies along the video, fps treats those frames as happening at varying time intervals. 

An input video with PTSs starting past zero might yield unexpected results. Suppose the input PTSs start at 300, and say that converts to 10 seconds. Then fps repeats the first frame to fill the first 10 seconds of the output video. (However, ffmpeg may suppress those repeated frames, depending on the -vsync setting.) If you set start_time to 10 seconds, then fps sets the sync point to the PTSs converted from 10 seconds on the timeline. It no longer repeats the first frame. And, it starts the output PTS at a value corresponding to 10 seconds, instead of zero.

Examples

  • A typical usage to make a video with a frame rate of 25 frames per second, from an input video with any frame rate:
    • fps=fps=25
  • The output frames have PTSs of 0, 1, 2, etc. The frame rate is 25/1. The time base is 1/25.
  • Make a video with a frame rate of 24 frames per second, using an abbreviation, and rounding method to round to nearest:
    • fps=fps=film:round=near
  • Clean up a video with varying time between frames, and dropped frames. The input video is supposed to have an NTSC standard frame rate of 29.97 frames per second, and the time base is 3003/90000, but the PTSs increment variably at slightly more and less than that rate. The recorder dropped some frames, but the PTSs still reflect when the remaining frames were captured. 
    • fps=fps=30/1.001:round=near
  • The output PTSs are 0, 1, 2, etc. The time between frames is exact. The output frame rate is 30000/1001. The time base is 1001/30000. Where frames were dropped by the recorder, fps repeated frames to fill the gaps.

Two musicians, violating social distancing

Posted by on 31 Mar 2020 | Tagged as: community, music

I was annoyed! Today I caught two musicians together. They were violating social distancing. I heard them counting: one was saying “1 2 3, 1 2 3,”…; the other, “1 2 3 4 5, 1 2 3 4 5,”… Actually, their “1”s landed together, so it sounded more like “ONE two twothree fourthree five ONE…”. Anyhow, we all need to be virus police these days, so I confronted them — from a safe distance, of course. “Hey!”, I said, “There’s a pandemic. Don’t be so close together! It’s unsafe.” “No worries”, the musicians replied, “we have two meters between us.”

Happy new decade, yes, 2020 not 2021

Posted by on 31 Jan 2020 | Tagged as: culture, robobait, time

Happy new decade, everyone! This month, January 2020, marks the start of a new decade, the 2020’s. And for those of you who are saying, “no, strictly speaking the decade doesn’t begin until 2021”, you are wrong. Or, at least, all of us who say that decades and centuries start on round numbers are just as “strictly speaking” right as you. Here is why.

I think we all agree that a “decade” generally means a 10-year time span, and a “century” means a 100-year time span. In everyday usage, the word “decade” means a time interval which starts on a year ending in “0”, and ends on a year ending in “9”. Thus we have just finished the “2010’s” (10 years from the start of 2010 to the end of 2019), and are entering the “2020’s” (2020–2029). By the same token, every-day use of the time unit “century” means a time interval which starts on a year ending in “00” and ends on a year ending in “99”. The previous century is known as the “1900’s”, and also as the “20th century”.

And all of the above only applies to English usage of North America, when using the current standard Gregorian Calendar. It is also limited to year 1 onwards — the Common Era (earlier labelled as Anno Domini or A.D. by Bede back in the 600’s and 700’s).
The pedants will point to the first year of the Common Era, which is year 1 rather than year 0. They argue that the first decade must begin with year 1 and be 10 years long, so it must end on year 10. Count forward in 10-year increments, and the 2020’s must start with the year 2021 and end with the year 2030. Similarly, the pedants argue that the first century must begin with year 1 and be 100 years long, so it must end on year 100. Note that the entire foundation of their pedantry is that the time unit “decade” must always and only be 10 years long, and “century” must always and only be 100 years long.

I believe that where the pedants go astray is to prize a constant length over a convenient starting point for “decade” and “century”.

Decades, and calendars, are social constructs. They don’t have to abide by strict mathematics, and they don’t. There is a value to having a calendar which matches the earth’s rotation around the sun. There is value to having a calendar which adjusts to seasonal changes in sunrise and sunset. And, there is value to having a calendar which matches common and convenient language usage.

Consider the day: normally it is 24 hours long. But when “springing forward” from Standard to Daylight Saving Time, there is a day which is 23 hours long. Later in the year, when “falling back” to Standard Time, there is a day which is 25 hours long. I have had to write software which handled this variation. A day is normally 24 hours, but can be 23 or 25 hours.

Consider the minute: normally it is 60 seconds long. But when a leap second is necessary — to keep the highly-accurate Coordinated Universal Time (UTC) in sync with the Earth’s variable orbital duration, and to keep the March equinox close to March 20 — the minute containing the leap second is 61 seconds long.

Some years are 365 days long, others are 366 days long, because of leap years.

Britain officially started its civil year on 25 March, until as late as 1751. Only in 1752 did it finally change the start of the civil year to 1 January. Common usage was ahead of civil rules in adopting 1 January as the beginning of the year, leading dates from 1 January to 24 March to be written with two alternative years: “30 January 1648/1649”.  Even more interestingly, September 1752 was 19 days long in England, instead of 30 days as in other years. This was to bring the calendar back in sync with the solar year, or in other words, to move March 20 back to the day of the March equinox.

All these calendar shenanigans are about deciding to allow social (and astronomical) considerations take precedence over strictly consistent numerical duration. “Strictly speaking”, a calendar balances all these considerations. And rightly so.

Given that, I think it is perfectly reasonable to declare: decades are normally 10 years long, but the first decade of the Common Era was 9 years long: 1 to 9 CE. 10 CE began the second decade. Centuries are normally 100 years long, but the first century CD was 99 years long: 1 to 99 CE.

Thus, 1 Jan 2020 CE also began a decade. Pedants, I see where you are coming from, but if you are willing to use a calendar with 25-hour days, and a 19-day September back in the day, I claim you cannot justify denying the 9-year decade and the 99-year century.

I am tempted to find a time machine, go back to northeast England in the 7th or 8th centuries CE, and persuade Bede to define his Anno Domini year numbers at 0, instead of 1. It would sidestep this whole decade and century problem. The only problem is, that in Bede’s time, Europe would not learn of the zero for another 3-5 centuries (where “centuries” are not strict 100-year time durations).

Happy new decade, starting January 2020!

Impressions of Italian grocery stores

Posted by on 31 Oct 2019 | Tagged as: culture, personal, travel

For the last six weeks, I’ve been shopping at grocery stores in Venice, Naples, Rome, and Florence. My Carissima and I are self-catering during an extended trip through Italy. My grocery store sample is limited and unrepresentative, but even so, the experience makes me appreciate the treasures at these Italian groceries. And, it helps me appreciate by contrast what they grocery stores back home in B.C. do well. Continue Reading »

In which Jim and Ducky appear in the Venice Biennale

Posted by on 30 Sep 2019 | Tagged as: culture, music, travel

A most extraordinary thing happened last Saturday: Ducky Sherwood and I appeared in the Venice Biennale! We were a (tiny) part of Sun & Sea [Marina], an “opera-performance” by Rugilė Barzdžiukaitė, Vaiva Grainytė, and Lina Lapelytė, at the Lithuania Pavilion. This work won them the Golden Lion award for Best National Participation. I have been a devoted amateur opera singer for nearly 25 years. I am delighted to join in this performance, in my little comprimario role. It is an unlikely addition to my résumé. Continue Reading »

Duplicate entry names in a single directory on a file server: solved!

Posted by on 31 Aug 2019 | Tagged as: robobait, software engineering

I have just seen — and solved — the most remarkable thing in a deep corner of my large archive disk: a single directory containing two entries (subdirectories) with the same name and same inode number. I will describe the problem, the diagnosis, and the cure for the benefit of others who encounter the same problem.

I was moving my archive of old files from one Network-Attached Storage (NAS) file server on my home network to another. Both old and new servers use netatalk AFP software to present Mac=style volumes to my Mac computer. Both run an underlying Unix-like OSs and file systems (but different ones for each).

I moved the archive by dragging the top-level directory data folder, using Finder on my Mac, from the old server to the new. Partway through, the copy aborted, with an error message like,  “a directory with the name .externalToolBuilders already exists”. This is remarkable. Each directory on the old server might have many entries or few, but each entry must has a different name. It is one of the fundamental rules of file systems. I was not combining two directories together, where an entry from one directory might collide with an entry with the same name from the other directory. Continue Reading »

Macbeth

Posted by on 30 Jun 2019 | Tagged as: community, culture, music, personal, Vancouver

I’m going to be in an opera! I am in the chorus of Heroic Opera’s production of Verdi’s Macbeth on Friday 5. July and Saturday 6. July in Vancouver. It will be a marvelous show. The singers are powerful and exciting, the direction is incisive, the costumes are lavish.

Continue Reading »

Good Godless Grief Songs

Posted by on 31 May 2019 | Tagged as: community, culture, music

I am on the lookout for good songs to sing at bad times. I want songs of grief and loss, suitable for amateur musicians like me to sing at funerals and memorial services, that do not mention gods, creators, heaven, or other fables. I am looking for “Good Godless Grief Songs”.

Continue Reading »

How to convert Google Docs to Markdown format

Posted by on 30 Apr 2019 | Tagged as: robobait, software engineering

Recently I needed to convert a Google Docs wordprocessing document to Markdown format (Github’s dialect). A simple web search turned up several hits, most of them unhelpful. I finally found a Google Apps script to do the conversion, which was almost, but not quite, suitable. But with a simple modification, it did the trick. I am sharing it here, in the hope that it will be helpful to someone else searching for “convert Google Docs to Markdown”.

Continue Reading »

For only the second time in 41 years

Posted by on 31 Mar 2019 | Tagged as: culture, Keyboard Philharmonic

January 1st each year is, among other things, Public Domain Day. This is the day, in most industrialised countries, when the copyright period expires on those works which became old enough in the past year. On Public Domain Day, those books, music scores, and artworks enter the public domain en masse. They are free for everyone to use and re-use without asking permission.

This year, Public Domain Day in the USA was notable. For only the second time in 41 years, works actually entered the public domain in the USA on that day. The last time this had happened was in 1998, and before that, 1977. These two 21-year droughts were the results of changes to US copyright law, first in 1976, and again in 1998. “The public domain has been frozen in time for 20 years”, quoted Smithsonian magazine.  Cultural advocates celebrated how the arrival of works into the public domain enriches culture generally in the USA. But they focussed more on literature. I am interested in music scores. Continue Reading »

Next »