/ Tech /  5 min read

Why use Linux for VFX?

When this question is asked, the question between the lines is really "Why don't they use Windows?" since the only other real contender outside Linux is Apple OSX, which itself is based on BSD Unix. For most intents and purposes, Unix and Linux are pretty interchangeable.

Only very small boutique shops and freelancers use MS Windows exclusively, and I have found that this confounds young artists new to the industry as well as veterans who have only ever worked at smaller shops. The ease of use and ease of intitial set up of MS Windows seem like home runs to a lot of artists, and so they are left scratching their heads. While I could focus on the historical reasons for Unix use , I would rather talk specifically about a few technical reasons why building a pipeline on Linux/Unix is relatively easy, whereas doing the same on Windows is downright painful, if not outright unfeasible. Perhaps I will encourage Aaron to make a post about the historical set of reasons for using Unix. When I say "Unix" from here on out, it also includes Linux, unless otherwise stated.

Unix has had solid support for symbolic links for decades. Windows has barely added this in Vista, and even then it is still not very good (and, no, shortcuts are NOT the same thing as symbolic links). Why would you want symbolic links you ask? Well, let's look at a toy pipeline for the answer (this will be massively oversimplified and is a poor pipeline model, but works for demonstration). Let's say our system is very simple: for a given shot in a show, we have a directory that holds all of our files (scene, geometry, textures, the current version of the plates from the client, etc.). This directory is a version number. So our directory looks somthing like this:

/proj/seq/shot
    001
    002
    003

Every time we want to version up, we copy the latest version and renumber it. Dandy! Now what is the problem that happens when our scene file (let's say it is Maya) is versioned up from v2 to v3? All the file paths still point to v2! Now your supervisor is going to be angry because you didn't update your geometry of that one creature for your scene. "But I did, I swear I did!" you screech in desperation. Well, it doesn't matter that you updated the geometry file in version 3, because your scene file was still pointed to the version 2 geometry file. This may seem like an easy fix to do by hand, but it gets irritating once you are on version 50 and still doing it over and over and over again. Also, once you are referencing dozens or hundreds of things into your scene, it simply becomes unfeasible to do by hand anymore.

So, what is a simple solution? We write a simple script for versioning up. It does the copy operation we previously did by hand, but it also adds one more thing: a symbolic link called "current". So now our directory looks like this:

/proj/seq/shot
    001
    002
    003
    current -> 003

And we always work out of current. We make sure all of our referenced file paths point through current as well. If we need to go back to a previous version, we make sure to change the "current" sym link to point there first. We would probably do this through some sort of "job in" script so, again, we wouldn't need to do this by hand each time. Again, this is a simplified example, but these sorts of things happen all the time in a real production pipeline.

2. Atomic file renames

By atomic, I mean the operation either fails or succeeds. There is no in between state; the file can't be partially renamed. Along with this, renames will overwrite other existing files. This is a dual edged sword since you do need to be careful, but you have the power to do so when you need it.

Lets use python to show this example. In Unix systems, if I want to rename a file, all I do is this:

import os
# I don't care if the destination file exists, just clobber it
os.rename("some_temp_file.obj", "final_file.obj")

This previous command will only fail if I don't have access to the file system where these file(s) exist, or I don't have permission to modify them. But, barring that, it will work.

If you try to rename over a file that already exists in Windows, it will fail with an error, no matter what. Thus, renames cannot ever be truly atomic. The only hope on windows is to do one of the following (neither of which is very good):

First solution:

import shutil
# uh-oh, our file is corrupt if we fail partway through
shutil.copy("some_temp_file.obj", "final_file.obj")

Second solution:

import os
# I sure hope this OTHER temp file doesn't already exist
os.rename("final_file.obj", "some_other_temp_file.obj")
# someone else could still recreate this file before I have a chance to rename it.
# Better known as a race condition. In windows, it is unavoidable in this situation.
os.rename("some_temp_file.obj", "final_file.obj")
os.remove("some_other_temp_file.obj")

3. Powerful built in file permissions

Yes, yes, windows has solutions for this using access lists and other such things when doing network logins, but they are far removed from the average or even the technical user, which is most VFX artists. Also, they are not a built in part of the OS. Windows was not built with networking in mind; it was bolted on after the fact and it shows. Windows was not built as a multi-user OS; again, bolted on after the fact and it shows. Unix has always been multi-user since the very beginning. Networking became a part of Unix before Windows (or MS DOS for that matter) even existed. Thus, both are extremely mature and fairly standardized.

4. Mount points for networked drives can be anywhere

On windows they need to be on a lettered drive, or using a network share mount. On Unix, you never need to even know it is a network mount. Thus it can easily be swapped out later without totally breaking things. With pipelines that are ever evolving, but still need to maintain some backwards compatibility, this is a huge boon. As a case in point, during university, I worked at a small shop on campus that used Windows. At one point, the IT folks at the shop decided we all needed to move from the S: drive to the K: drive for our network shares. Needless to say, pretty much all of our scene files broke.

Conclusion

There are other things that make Unix easier to work with in production pipelines, but the above are the biggest ones IMHO. Hopefully some of the appeal is a bit clearer to you now than it was before. If not, send an email to info@metapipe.com. The message should make its way to me and I'll do what I can to answer your questions.

UPDATE 2018-04-06: Added link to Aaron's Unix/Linux history in VFX and Animation post