Pipeline Movies

The pipeline is always with us. Even if we don't think about it, because it is different from studio to studio, it is exciting to see how they deal with it.

Old but gold (actually, Rhythm & Hues no longer exists, which is the most depressing story of all time in the industry).

Rhythm & Hues - A Framework for Global Visual Effects Production Pipelines - SIGGRAPH 2014 from Murali Anagani on Vimeo.

Since the ShotGun Software has been renamed to ShotGrid, which is a more PC name for a production tracking system :)

Tony Barbieri and Don Parker; The Shotgun Pipeline at PSYOP from Autodesk Media and Entertainment on Vimeo.

The best pipeline presentation(s) of all time (still ancient which is a bit of a sign that nowadays there are less intersting talk on that field).

Not soo long time ago there was a discussion hosted by Side Effects Software at Siggraph.

Or I'm just getting old and only have good memories from the far past—cinesite's presentation about generalizing pipeline and implementing fTrack.

Last but not least: if we are talking about the pipeline, it's unavoidable to talk about Katana. The exciting part is that people may think that Katana is a lighting & rendering tool. Actually, it is a pipeline tool because of the nature of rendering. If you want to render anything, you must create a complex system to flow the data from DCCs (digital content creation tools) to the render engine. That's where Katana comes in. So here is a brilliant presentation about Katana deployment.

Naming Convention 3. - Files and Folders 3.

To continue the previous chapter we have to further analyze the structure of a VFX production.

201401_OBP/
    010_in/
    020_preProd/
    030_assetProd
    040_shotProd
    050_out/

Just to mention there could be a folder hierarchy based on kind of a workflows like, edit, paint, composition, or hybrid ones like 2D, 3D. I not recommend that because of redundancy and the lot of time we have to spend to jump for eg. from 2D to 3D or paint and composition since we usually work on one shot at time. There is an exception:

Edit
Editing as a workflow is an exception because this is the workflow rule them all. So it should be at the top level of the structure. A VFX studio usually not responsible for the editing, but it is very important to have the ability to watch to VFX shots in context. So there have to be editorial information, pre-cuts or rough-cuts, sequences in together, etc. Huge topic so might discuss it later. The conclusion is the following:

201401_OBP/
    010_in/
    020_preProd/
    030_assetProd
    040_shotProd
    050_edit
    060_out/

In
Before we get to the fan part, it is time to consider that I tend to call secondary stuff like documents, storyboards, audio, references, etc. Where to put them. For the in folder the definition was: things which was not created in-house during the projects. Being a VFX studio it is most likely that all these type of things should be placed under the in folder. For a small production I do not recommend to use sub-folders (flat vs deep structure - flat is the winner I think), but for this case I use sub-folders to demonstrate it.

  201401_OBP/
    010_in/
        010_script-storyboard
        020_documents
        030_references
    020_preProd/
    030_assetProd
    040_shotProd
    050_edit
    060_out/

Another very important rule for the in folder: never (never never) reference files from here. If you see somebody referencing an image for eg. in a nuke script you should fire he or she or ask your boss to do that. And of course you should be promoted being recognized that.

Naming Convention 3. - Files and Folders 2.

To continue the previous chapter the starting point is the following stucture (I removed postprod folder explained later):

201401_OBP/
    010_IN/
    020_PREPROD/
    030_PROD/

    040_OUT/

When we start to create directory structure we are about to define the pipeline as well. By doing that we define concepts and create rules. Let's think about folders as a sets. We have a lot of things in the VFX production and we analyze those to define what is what. Does the particular item fit into that set or not? For example: is the current 3D model an asset or a shot. The answer is quite clear. Is that model a preprod(uction) or prod(uction) model? Not sure. To make it clear we have to create a rule like the following: there is no preprod model because preproducion is only a step its creation process ( for eg.: version 003 is more like preprod asset but version 008 is hires production model).

So right now we have to define a couple of concepts, pre-production (preprod), production, asset and shot produciton.

Pre-production
Less managed workflows like concept art, previz, animatics, storyboard. In VFX production it is usually before the shooting so it means there are no shots. Actually if we make previz or animatics we create shots but they can't be the production shots (except for full CG shots - see below).

Production
It is important to note that in VFX Point Of View there is no post-production. The whole film making process of course has post-production process. But if we use prod and post-prod folder one of them is kind of a meaningless. I hope it is not confusing.
So in my definition production for VFX is the process on the shots created by principal photography.

Asset production
Asset is a general concept in VFX. There is the concept asset production. Basically it means we create things which can be used in multiple shots, or we produce things shot independently.
Usually we are talking about sets like: characters, sets, enviroments, props.
 

Shot production
Shot is a general noun in the film industry. In a context of VFX shot production means obviously the creation of the VFX shots for the film. It can be divided to VFX shots, full CG shots and so called post FX shots (usually simple 2D effects like translate or rotate the image).

For shot production
Pre-producion can be defined like this: if we working on shots before we get shots created by principal photography it has to be the pre-production.

So asset and shot production basically a production and should below the 030_PROD folder.

201401_OBP/
    010_IN/
    020_PREPROD/
    030_PROD/
        ASSETPROD
        SHOTPROD
    040_OUT/

But can be simpler like this (to avoid deeper folder structure):

201401_OBP/
    010_IN/
    020_PREPROD/
    030_ASSETPROD
    040_SHOTPROD
    050_OUT/

You may ask: what is the logic behind using capital letters. And the answer is: fashion. There is no reason for that. So it should look like this using camelCase:

201401_OBP/
    010_in/
    020_preProd/
    030_assetProd
    040_shotProd
    050_out/


To be continued...

Digital Production Pipeline

Literally the best video on youtube. I think don't have to append anything to this.

Naming Convention 4. - Numbers

We use numbers to count things. Avoid to use numbers on distinguish things.
Typical example to count versions like:
v001
v002
...
But not the best practice to distinguish different things like:
Tree_01
Tree_02
I usually use letters to distinguish variations.
Tree_A
Tree_B
...
Tree_Z
Using one letter can refer to 26 variations. We can extend it with double letters like:
Tree_AA
Tree_AB
...
We can combine variation with counter like this:
Tree_A_01
Tree_A_02
Tree_B_01
Tree_A_02
...

Anyway, this topic is about using numbers. We are talking about CG and VFX (as TidyVFX is about that) but this knowledge is important for the whole universe of course :)

To establish a naming convention means we have to specify for what to use numbers.

1. Count things
As I wrote above

2. Arrange things
For files and folders usually have to use numbers to specify a certain order because file systems use alphabetical order by default. For eg. if we have IN, OUT, PROD it would look like this in the file system:

But if you want a certain order you have to use numbers like this:

2.1 Padding series
Because of the file systems another thing to deal with is the padding. It is quite simple.
One character can produce ordered sequence from 1 to 9, two character from 1 to 99, etc.
For image sequences in VFX we usually use 4 digits and that means correct order 1 to 9999.
Also have to mention never start a sequence from 1 because it is a big issue when we have to extent it with 80 frame at the beginning for some reason. It would start from -80 and we really don't want that.

2.2 Padding shots (insert shots)
There is another function of padding in shot numbers for eg. We have to consider there could be new/extra shots during the production. So if we use 001, 002, 003, we can't insert a shot between 001 and 002. We have to extend the name with zeros. Same rule. If we use one character that means we can insert 9 extra shot.
So the shot numbers should be like this:
0010
0020
0030
The rule for inserting extra shots is to halving the numbers. To insert shot between 0010 and 0020 would be 0015. To insert a shot between 0010 and 0015 would be 0013 (or 0012).
I would not recommend to use 001A, 001B for shot names.

Naming Convention 3. - Files and Folders 1.

Let's pretend we have a project. A good starting point is to count how many things we have to deal with during a project. Obviously it could be tons of files and data. There are a lot of ways to deal with them but eternal question is "what is what?". So we have something and we have to give a name for that. Considering this we have to define what this something is. What is the reason for this something? For what we are going to use it? Etc.
To answer these questions we have to have a look at our project. That something can be an image file of a concept art or a 3d model or it can be a feedback that our client sent us.
Of course every project is different so we can only talk about it in general. This blog is about VFX-CG so I try to collect ideas about that kind of projects.

First of all we know there are inputs and outputs.
Inputs can be any kind of material which is not created during the project in-house.
Outputs can be any kind of material which is sent to the client, co-workers, off-site studios, etc.

If we are talking about folder structure it starts like this:

IN
OUT

One step back. There should be a solution to identify projects (or shows).
Date is a simple organizer with a project ID.

Definition:
[yyyy][counter][separator][ProjectID]
[yyyy] - year like 2014
[counter] - two digit like 01
[separator] - underscore
[ProjectID] - abbreviated name of the project - exactly 3 capital letters like OBP

For eg.:
201401_OBP (Obviously Basic Project)

So we have:

201401_OBP/
   IN/
   OUT/

Of course we shoud have something called production. We can extend that concept with pre-production and/or post-production.

That is the (first) point where we have to consider importance or priority because we have options to organize the folder hierarchy and there is no ultimate solution as I wrote before. One really important thing is to try to avoid redundancy. For example we can arrange folders like this:

201401_OBP/
    IN/
    PRODUCTION/
        PREPROD/
        PROD/
        POSTPROD/
    OUT/

As you can see PRODUCTION or PROD basically the same but there are two folders with almost the same name. So we can avoid that like this:

201401_OBP/
    IN/
    PREPROD/
    POSTPROD/
    PROD/
    OUT/

Usually this is also called flat vs deep directory stucture.

A typical file system arranges folders like above therefore we should use numbers to add a precise order of the folders like this:

201401_OBP/
    010_IN/
    020_PREPROD/
    030_PROD/
    040_POSTPROD/
    050_OUT/

About using numbers you can read this: Naming Convention 4. - Numbers

To be continued...

Naming Convention 2. - Pipeline (CG / VFX)

When we are thinking about naming convention more or less thinking about pipeline. So couple of words about pipeline.

We can separate workflow and pipeline. I could write a novel about this. Just look these words up in wikipedia. Of course we are talking about VFX or CG here so the simple conclusion is the following:

Workflow
A process like modeling, riggin, texturing, etc.

Pipeline
There is now exact wikipedia article about VFX pipeline. In general it means "chain of data processing stages". In VFX it could be workflows and their connections. We can imagine some kind of flowchart like this:

The arrows can be simple data input/out put connections or data processing as well. The boxes are the workflows as you can see above. Pipeline can refer to the arrows. Personally I prefer the concept that pipeline is the whole stuff (workflows and the arrows) as you can see above.


With computers we create, modify and store data. As simple as that. So the question is how we do that?

Workflow means (most of the time) we use softwares like Maya, Nuke, Houdini, Blender, Photoshop, etc.. They have input and output data: usually files. We have to store them in a consistent manner. Naming convention refers to that consistency.

We have to separate files from each other based on the content and store information within the filename. To organize files we use folders.
So the goal to establish an organized folder and file structure ruled by the naming convention.


Files and folders are ruled by the operating system (OS) therefore we have to accommodate these rules to the naming convention.

As I wrote before there are couple of problems with of folder and file structures in general. For example:
1. Limitation of the length of the file path.
2. One file can be only in one folder.
But this would be another topic. I'm going to write about it.

Naming Convention 1.

Simply it is the most important weather you are an artist or technician (so called TD) in the VFX industry. Furthermore it is the most important thing for mankind (or the universe). Naming convention is essential for everything.
Unique projects has unique naming convention. But let's pretend you won't do anything that nobody did before meaning of the process. So there are guidelines.
Naming convention is extremely important but there is another principle. As Richard Williams wrote in his book: "Keep it stupid simple." aka KISS.
Working with computer means you use operation systems like Windows and you create files which could be organized in directories. But you will find out sooner or later there are at least 2 big problem here.

Big problem 1.
One file can only belong to one directory.

Big problem 2.
More data we have that we can simply put into names, filenames for eg.

But these problems raise a lot of other issues. And this blog is more or less about to discuss those issues. Right now I simply use the KISS principle which also means (for me) do it as you can.

So we need a general solution of file naming. Before I tell you the truth I gather some ideas.

1. Importance of the order

Usually files are arranged by their names (alphabetical order).  So by giving names we can describe the order of the files.

2. Parts (tags)

We can of course combine different information usually called tags. So the question is how many tags should be there to collect the most important information?

3. Compressing information

We can say abbreviation but is more than that. Question is how can we compress more and more information into alphanumeric character. So for eg. we can use abbreviations but in that case we should establish a dictionary for those, because BG does not necessarily mean background for everybody.

4. Documentation

If you are a one person studio you might think documentation is not important. But it is. I think it is for every scale of facilities. So the naming convention should be documented. It can change over time from project to project and that means the naming convention document can have versions or can be attached to projects. It could be a simple text file.

4.1 Documentation template

Usual form to template naming convention is:
[tag1][separator][tag2][tag3][separator][tag4]

It can describe a file rules as well.

We can use explicit characters and variable names to be more clear.
[name]_[versionLetter][versionNumber]_[comment]

It could be necessary to explain the rule of the variables. We can call it file rules.
[name] - Starts with capital alphabetical character and contains any alphanumeric character
[versionLetter] - Any non-capital alphabetical character

This example above use the "_" (underscore) as a constant character or tag. That means you can't change it. Usually when we describe a file path there are constant directories so they are constants as well:
e:/ Projects / [ProjectID] / 2D / ConceptArt / [AssetID] / 
[AssetID]_[versionLetter][versionNumber]_[comment]

In this example above the "Projects", "2D" and the "ConceptArt" are constant tags.

More complex when we use expressions like "AssetID", because this is not a simple variable but a concept plus and abbreviation (ID = identification). We should describe things like Asset and/or AssetID before we use it. But it is not the part of the naming convention documentation rather a documentation of the pipeline or system we utilize.

5. Character table
It is always recommended not to use special characters. That means file names contain only alphanumeric character, underscore and hyphen. We can express with regular expressions: [A-Za-z0-9_-] (see below)

So this would be my general template for file naming:
[name]_[vatriationLetter]_[versionLetter][versionNumber]_[comment]

And the rule for this pattern:
[name] - Starts with capital alphabetical character and contains any alphanumeric character
[variantLetter] - Only one capital alphabetical character
[versionLetter] - Only one non-capital alphabetical character
[versionNumber] - Two digit starts with 01
[comment] - Any alphanumeric character


Below we can see a series of examples how it looks in practice. So it is more like the explanation of the concept behind this naming convention.
Let's say I'm working on a landscape concept art. So the name part would be "Landscape". I separate the process stages like sketch, details, colors. So the version letters represent these stages (a, b, c). I can also comment where the stage starts to be more clarify.
Usually when we work on something we don't consider at the beginning there might be another variant for the subject. So if we use "A" as a base variant we won't ruin the naming convention later with adding an extra tag.

Landscape_A_a01_Sketch
Landscape_A_a02
Landscape_A_a03_Background
Landscape_A_a04
Landscape_A_a05_FGTrees
Landscape_A_b01_Details
Landscape_A_b02
Landscape_A_b03_BGMountains
Landscape_A_b04
Landscape_A_b05
Landscape_A_c01_Colors
Landscape_A_c01_SkyAndClouds

You might find the "a", "b", "c" a little bit confusion to use for versions. You are not the only one. Most people use a "v" letter like  "v01", "v02", etc. But using only one constant version letter we can't embed more information with that.


Footnotes

Regular Expressions aka Regex

Regex is a tool to analyze names. Usually it used to verify and search. If you have a basic idea how regex works you might want to establish naming convention that can be easily handled with regex.
For eg. the naming convention above can be described as regex pattern like this:
[A-Z] [A-Za-z0-9]+_[A-Z]_[a-z][0-9]{2}_[A-Za-z0-9]+

So the regular expression tells to the engine the name:
Starts with one capital alphabetical character [A-Z], followed by alphanumerical character [A-Za-z0-9], it could be one or more +, followed by an underscore _ (constant letter), followed by a capital alphabetical character [A-Z], followed by an underscore, followed by a non-capital alphabetical character [a-z], followed by any number [0-9], it has to be exactly two digit {2}, followed by underscore, followed by any alphanumerical character [A-Za-z0-9] and it could be one or more +.

The real benefit to use regex comes when you or your studio use programming languages or other tools to verify or match paths, file names, or any kind of text, etc.