Sep 12 2013

Mango Version Viewer & Dependecy Tracking

I’m happy to share with you guys a screen grab of the dependency viewer for “mango version viewer”.. view the versions associated with any element being reviewed any time any place with out out having to open the work-files them self’s
dependencies are hierarchical so that you can easily view them in a natural way…

 


Sep 1 2013

Mango Phase 1

Some time has passed since my last Mango related post. Thankfully, I have made significant progress. I’m glad to announce that Mango Phase 1 will be completed in the next few weeks. There are three important developments which I will address in this post: resource-tracking, the Software Launcher, and user applications and tools (stand-alone and software-integrated).

 

Mango’s Resource-Tracking

Mango Phase 1 has been about setting up the structure on which the entire pipeline is based. I implemented base APIs that will support all of the core stand-alone applications and software-integrated tools. The two most significant APIs in this stage of development are:

  • versionTracker (API for versioning of resources with database backend)
  • namingAPI (API for generating the folder structure that host the files that make up the files associated with versions made by versionTracker)

These two main components come together to create a resource-tracking system. In Phase 1, the development of the resource-tracking system has been focused on the needs of smaller studios in rapid production environments. This focus allows smaller studios to start using resource-tracking without being overly intrusive or creating a bottleneck for the artist by over-departmentalizing the process. To achieve this implementation, Mango Phase 1 concentrated on establishing the standards and practices necessary for creating and tracking the main resource types that are the most important in the “generalist” environment. In particular, I have focused on artist “work files”, “render elements”, and “shot plates.” In Phase 2, I will move toward more detailed resource-tracking and implement new resource types for “models”, “rigs”, “animation caches”, “cameras,” and “shaders.”

Mango’s Software Launcher

The Software Launcher is the common entrance to the pipeline. It is a stand-alone application that allows supervisors to control the software configuration by creating unique profiles for their projects. These profiles control which software, plug-ins, third party tools, and extensions are used by artists on their projects. Profiles allow artist to jump between projects, while guaranteeing that every artist under the same project has the same set of tools. Moreover, this profile mechanism allows supervisors to adopt new software or plugin versions without unintended negative effects on concurrent projects.  The Software Launcher and its profile mechanisms means that studios no longer have to rely on IT to push plug-in installations over large networks via more traditional time-consuming means that can be unreliable (e.g., MSI, Psexec, manual installations).

The connection between Mango and the artist’s software relies on the modular plug-in distribution system of the Software Launcher. Mango and its tools get attached to the target application via the “LosPipeLine” Software Launcher module which should be part of every Software Launcher profile. The Software Launcher also offers and optional command line mode. This mode is used to apply the profile-based software configuration approach to network rendering solutions, such as Thinkbox’s Deadline.

User Applications ( stand alone and software-integrated)

Additionally, the deployment of Phase 1  focused on the “hand off” of  resources across the different disciplines of the shot creation process (e.g., how a lighter hands off lighting elements to a compositor). Mango is composed of several applications and tools, both stand-alone and software-integrated. These tools are easy-to-use interfaces for artists, supervisors, and managers to facilitate the adoption of the Mango workflow. Here’s a list of the initial set of applications and tools that are bundled with Mango Phase 1:

  • Project Manager (stand-alone)
    • Simple interface for quickly creating the base directory for a new project
    • Allows for adding sequences and shots at anytime to any project

 

  • Workspace Manager (Integrated with 3ds max, maya, and nuke)
    • Simple interface for creating and navigating artists’ work areas, work files, and snap shots

  • Render Element Farm Submission Tool (Integrated with 3ds max via custom render pass manager)
    • Integrated element publishing and dependency tracking

  • Comp Central (Integrated with nuke)
    • Interface for browsing, importing, and updating render elements, plates, and cameras related to a shot directly from nuke

  • Element Publish (Nuke)
    • Publish comp, precomp, paint resources out of nuke
    • Automatically configure and version up your read nodes
    • Local or farm (via Deadline) publishing

  • Version Viewer (stand-alone)
    • Tool for quickly listing, navigating, and filtering all resources and their versions for a particular project, sequence, or shot

  • Plate Ingestion Tool (Integrated with Version Viewer)
    • Quickly publishes plate resources into the pipeline along with any other auxiliary data (e.g., camera information, hdri data, lens distortion, and LUTs)


May 23 2013

“Mango” Production Tracking System + Full Service Pipeline (PTS)

What is PTS?

A PTS (Production Tracking System) is a database that manages VFX/Animation projects and studio resources. This system helps producers, supervisors, and artists keep track of the multiple tasks that are required to successfully complete a show. It facilitates communication between departments and provides real-time reports on the current progress and cost of work. The PTS represents the integration of a “back-end” database (e.g., postgreSQL or MySQL) and “front-end” applications. The everyday user interacts with “front-end” applications which are designed to meet his unique needs. These applications can be stand-alone or integrated into third party commercial or proprietary software (e.g., task management application or resource publishing tools inside of Maya, Nuke, etc.).

Why is there a need for a PTS?

A well-designed PTS is integral to an effective studio. It allows artists, supervisors, and producers to find relevant information quickly. For example, an artist can see a list of incomplete tasks that are assigned to him. Then, a supervisor can use PTS to comment on the tasks which artists marked “ready for review” and keep a historical log of these comments. While a producer can use PTS to see which shots are over-budget and the human resources department can have a record of billable hours that are connected to specific tasks. Everyone involved in the production process is able to input and query this information through applications that are stand-alone and/or integrated with third-party commercial or proprietary software.

A few examples of common stand-alone applications in PTS are:

  • A Time-Card Application
    • creates an electronic record of billable hours in the database which replaces paper time-cards.
    • allows artists to connect these hours to a particular task which can be used to create more detailed reports.
  • A Task Management Application
    • allows supervisors to break large scale work into smaller easily-delegated tasks.
    • holds other important information to facilitate time-management (e.g., priorities and due-dates).
    • connects tasks to progress-tracking statuses (e.g., “in progress” or “awaiting supervisor review) which can be filtered by supervisors and artists.
  • A Dailies Application
    • lists all the image sequences sent by artists to be reviewed in dailies.
    • uses the versioning system in PTS to help supervisors quickly find older versions of the same sequence for comparison during review.
    • connects comments about the work being reviewed in dailies to the task that generated it to ensure the artist has a record of the notes.

A few examples of common applications which are integrated into third-party software in PTS are:

  • Publishing and Versioning Tools
    • help artists version their work while keeping a historical record of changes.
    • allow for easier hand-off of versioned work to the next step in the pipeline.
    • when necessary, allow artists to easily roll-back to previous versions of their work.
    • adds a layer of programmable quality control to ensure smooth ingestion into the pipeline.
  • An Asset Catalog
    • helps animators and layout artists quickly find and reference show-specific and updatable set-ups.
  • A Shot Resource Catalog
    • allows lighters and effects artists to quickly know the resources that make-up a 3d scene.
    • allows artists to compare version information about the resources currently in the scene.
    • Alerts artists when scene resources need to be updated.
  • Scene Building Tools
    • Eliminates the need for multiple artists to work on the same file by allowing artists to build unique files with shared resources.

A PTS adds accountability and power to a studio because it automates some of the more tedious aspects of the VFX/Animation pipeline. A pipeline that revolves around a well-designed PTS allows artists to focus the majority of their time on art, rather than addressing technical complications that arise from keeping scene files updated. In addition, it gives supervisors and producers the real-time information that allows them to track the progress of work and make better choices which maximize the project’s profit.

Why is a PTS not as Common in Smaller Studios?

If a PTS is so wonderful, why is everyone not using one? There are four simple answers to this question:

  1. Smaller studios don’t usually have the resources to employ full-time programers. So, developing and integrating a PTS into their pipeline is not cost-effective.
  2. The culture of many smaller studios means that there are no uniform standards and practices about how scene files are created and shared. Therefore, their database (the “back-end” of a PTS) mirrors this complexity and quickly outlives its usefulness.
  3. Popular expensive commercial solutions (e.g., Shotgun/Tank, FileMaker Pro, SharePoint) are a framework for a database but, they don’t offer uniform standards and practices. This leads to the previously described problem for studios who do not have strong standards and practices. Moreover, many of these commercial solutions don’t offer third-party integrated applications leaving studios to write their own.
  4. These expensive commercial options still require the additional cost of Technical Directors to update, maintain, populate, and further develop these systems; sometimes greatly increasing the cost of integrating such a system.

While smaller studios have a very large incentive to invest in a pipeline that revolves around a well-designed PTS, they often don’t. The expense in time and money for an inadequate product is too much of a risk.

What is Mango?

Mango is an economical plug-and-play solution that is designed around how people actually work in VFX/Animation studios. It recognizes that many studios do not have set standards and practices which are a predictable and repeatable way of creating quality work (Modular Asset Pipeline Flow). Mango not only provides a uniform way of accomplishing this work, but it is a ready-to-use PTS that is modeled around a uniform set of standards and practices. In addition, Mango includes tools that will run on multiple applications (e.g., Maya, 3ds Max, Nuke, Houdini) to help enforce the uniform standards and practices without limiting artists creative ability.

Mango eliminates the need to have multiple programmers to mold the system to work in your studio. It greatly simplifies the nature of the database that you will use to track this work. Mango minimizes the footprint that these kind of systems can have in a studio pipeline. It fundamentally changes the way the studio works. In other words, Mango helps to create and monitor a more efficient studio work system by providing a full end-to-end pipeline.

When will Mango be Complete?

Mango is an on-going personal project on which I am working full-time from home. It is the result of 10 years of experience working in a range of studios in the VFX/Animation industry. The base for the PTS and several applications will be complete by the end of July. The development cycle is going to be rapid this is why I am researching tying together multiple open-source libraries and technologies (e.g., Python, postgreSQL, SQLAlchemy). Not only will this make my development cycle more efficient, but it also makes it more accessible to other programmers.

Anyway stay tuned, I will add more posts related to my findings about database programing for the VFX industry.

Cheers!


Apr 26 2013

Modular Asset Pipeline Flow (Standards And Practices)

One of the main interests in my career has been developing a way in which I could optimize the process of creating and revising animated content in 3d through uniform standards and practices. My interest in this topic is rooted in my experience as a Character Technical Director (TD). A Character TD collaborates with many different departments. For example, he is the one that makes sure that models are technically sound for rigging. He also has to collaborate with animators to ensure that the animation set-ups are simple yet powerful enough for the animator to be able to achieve the desired performance. A Character TD interacts with lighting and effects to make sure that the animated performance transfers from animation department to the lighting department without breaking. He is usually in charge of adding more animated detailed to assets via simulations like hair and cloth. These are just a few of the responsibilities of a Character TD,basically, he has his hands in everything. Owing to their work responsibilities, most Character TDs frequently assume some of the responsibilities of pipeline development.

While I’ve been fortunate enough to work at studios that have had well-designed asset-based pipelines, I have worked at others that have failed to develop the standards and practices necessary to implement such a pipeline. I have drawn on these professional experiences in both kinds of studios to develop a specialization in writing 3D pipelines and developing this, often missing, set of standards and practices. The link below is an image of an up-to-date concept map of how assets should flow and can be developed through a pipeline. There are many benefits to implementing this kind of pipeline. A few of the most notable include:

  • Modeling, rigging, shading, lighting, effects, and animation can evolve parallel to each other rather than being linearly dependent. This monumentally increases the speed with which people can accomplish work by eliminating a known bottleneck.
  • It facilitates the tracking of different versions of a resource that form an asset and even allow roll-back. This ensures that artists always have the correct versions of resources on which their work depends.
  • It eliminates the need of transitioning from previs to postvis work. This avoids a common redundant step in many small studio pipelines.

This standardized asset flow allows for a very simple database model which is able to track and manage the work greatly simplifying the development of a PTS system (mango pts).

Hopefully, I will be able to blog in more detail about this soon.

Until next time!

 

Modular Asset Pipeline Flow

 


Jan 14 2013

Creating Menu’s in max

so very often TD’s look for ways to distribute their tools via menu’s. yet it has been my experience that using the code that’s documented can be really flaky seance any error half way through your code.. can make max go into to a weird state along with making your menu file corrupt all together (which means max my not start again for you until you fix this file). so while doing some work with manu’s i decided to put together a simple object oriented API that anyone could use to build this menu’s virtually with a limited number of commands..
the concept behind it is the new menu becomes a virtual object created by TD that he can append other items too.. once he is done appending this items he can commit his new menu to the ui, via the myMenu.commitMenu command.. pretty simple (i hope :D)

/*
example starts here
*/
filein @"C:\maxScript\libs\maxMenuBarApi.ms"
myMenu = ::maxMenuBarApi()



--make menu items
/*
makeMenuItem takes the following inputs
1)name to be used for the menu item..
2)category of the macro script it will be execute
3)varible name of the macros script to execute
*/

helloWorldMenuItem = myMenu.makeMenuItem "hello World Test" "0 Pixomondo Macros" "helloWorld"
helloPeterMenuItem =myMenu.makeMenuItem "hello Peter Test" "0 Pixomondo Macros" "helloPeter"
separatorMenuItem = myMenu.makeSeparator ()

menuI = #() --this will hold all the items we are adding to our menu..

append menuI helloWorldMenuItem
append menuI helloPeterMenuItem
append menuI separatorMenuItem

--adding sub menu to our menu
subMenu = myMenu.makeSubMenu "subMenu"
myMenu.addItem helloWorldMenuItem subMenu:subMenu
myMenu.addItem helloPeterMenuItem subMenu:subMenu
append menuI subMenu

--adding a nested sub menu
subMenu = myMenu.makeSubMenu "Nested sub menu"

menuA =  myMenu.makeSubMenu "menu A"
myMenu.addItem menuA subMenu:subMenu

menuB =  myMenu.makeSubMenu "menu B"
myMenu.addItem menuB subMenu:subMenu

myMenu.addItem helloWorldMenuItem subMenu:menuA
myMenu.addItem helloPeterMenuItem subMenu:menuA

myMenu.addItem helloWorldMenuItem subMenu:menuB
myMenu.addItem helloPeterMenuItem subMenu:menuB

append menuI subMenu

--let's add all the items to our ui now..
for m in menuI do myMenu.addItem m
--let's rename the menu
myMenu.setMenuName "My Menu Api Test"
--let's commit the menu
myMenu.commitMenu()

anyway the hope for this api is that TD’s can deploy menu’s faster, and with less code.. along with adding a separation of the building of the code, and the actual drawing of the menu’s.. in hope’s that bugs regarding the building of the data, will not corrupt any of the menu files of or put max in “strange limbo mode”

here’s what the API looks like..

/*
Max menu bar api
by Carlos Anguiano
*/
struct maxMenuBarApi 
(
	menuData, 
	fn dicGetKeys dic =
	(
		
        --spit the posible keys out  
        DNArray = dotNetObject "System.string[]" dic.keys.count  
        dic.keys.copyTo DNArray 0  
        out = for i = 1 to dic.keys.count collect (DNArray.get (i-1))  
		out
	),
	fn dicRemoveKey dic pkey =
	(
		if dic.item[pKey] != undefined then dic.remove pKey
	),
	fn dicSetProperty dic pKey pVal =
	(
		dicRemoveKey dic pKey
		dic.add pKey pVal
		pVal
	),
	fn getMenuName =
	(
		menuData.item["name"]
	),
	fn setMenuName newName =
	(
		dicSetProperty menuData "name" newName
	),
	fn getMenuItems subMenu:undefined =
	(
		if subMenu == undefined then subMenu = menuData
		subMenu.item["items"]
	),
	fn makeMenuItem itemName macroCat macroName  =
	(
		itemData = dotNetObject "System.Collections.Hashtable" 
		dicSetProperty itemData "type" "menuitem"
		dicSetProperty itemData "title" itemName
		dicSetProperty itemData "category" macroCat
		dicSetProperty itemData "macroName" macroName
		itemData
	),
	fn makeSubMenu itemName =
	(
		itemData = dotNetObject "System.Collections.Hashtable" 
		dicSetProperty itemData "type" "submenu"
		dicSetProperty itemData "title" itemName
		dicSetProperty itemData "items" #()

		itemData
	),
	fn makeSeparator =
	(
		itemData = dotNetObject "System.Collections.Hashtable" 
		dicSetProperty itemData "type" "separator"
		itemData
	),
	fn addItem item subMenu:undefined =
	(
		if subMenu == undefined then subMenu = menuData
		ar = getmenuItems subMenu:subMenu
		append ar item
		dicSetProperty subMenu "items" ar
		true
	),
	fn clearMenu =
	(
		mainMenu = menuMan.findMenu (getMenuName()) --see if the current menu already exists
		if mainMenu != undefined do -- if pixo menu exists then clean it out..
		(
			menuMan.unRegisterMenu mainMenu
			menuMan.updateMenuBar() 
			thePixoMenu = undefined
		)
		true
	),
	fn addItemToMenu def menu =
	(
		case def.item["type"] of
		(
			"menuitem":
			(
				theAction = menuMan.createActionItem def.item["macroName"] def.item["category"]
				theAction.setTitle def.item["title"]
				theAction.setUseCustomTitle true
				menu.addItem theAction -1
			)
			"separator": 
			(
				theAction = menuMan.createSeparatorItem()
				menu.addItem theAction -1
			)
			"submenu":
			(
				sb = menuMan.createMenu def.item["title"]
				theSubMenu = menuMan.createSubMenuItem def.item["title"] sb
				menu.addItem theSubMenu -1
				for i in def.item["items"] do addItemToMenu i sb
			)
			default:(format "% is not a valid menu item..\n" def.item["type"])
		)	
	),
	fn commitMenu =
	(
		mainMenuBar = menuMan.getMainMenuBar() 
		menuItems = getMenuItems()
		
		if menuItems == 0 then 	throw "There are no items in this menu to commit"
		
		clearMenu()
		
		newMenu = menuMan.createMenu  (getMenuName())
		for it in menuItems do addItemToMenu it newMenu		
		
		SubMenu = menuMan.createSubMenuItem  (getMenuName()) newMenu
		
		mainMenuBar.addItem SubMenu -1
		menuMan.updateMenuBar() 
	),
	fn _init_ = 
	(
		menuData = dotNetObject "System.Collections.Hashtable" 
		dicSetProperty menuData "name" "testMenu"
		dicSetProperty menuData "items" #()

		true
	),
	init = _init_()
)



Sep 12 2012

Exporting .pc2 files straight out of maya..

So recently I’ve been working on maya to max pipeline…
i ran into some issues with the built in cacheFile mechanism in maya so i decided that seance .pc2 is such a simple binary format that i could just write my own .pc2 exporter!

here’s what i came up with 🙂

ps. this heavily borrowed from the blender importer written by “Matt Ebb” http://mattebb.com

import maya.cmds as cmds
import struct

def frange(x, y, jump,):
    ar =[]
    while x < y:
        ar.append(x)
        x += jump
    return ar

    
def writePC2File(filename, shape, world=True, zUp=False, startF=0, endF=100, samplesPerFrame=1):
    sampleRate = 1.0/samplesPerFrame
    numPoints = cmds.polyEvaluate(shape,v=True)#len(me.verts)
    totalFrames = endF - startF
    timeRange = frange(startF, startF+totalFrames+sampleRate, sampleRate)
    numSamples = len(timeRange)
    
    headerFormat='<12ciiffi'
    headerStr = struct.pack(headerFormat, 'P','O','I','N','T','C','A','C','H','E','2','\0', 1, numPoints, startF, sampleRate, numSamples)
    
    file = open(filename, "wb")
    file.write(headerStr)
    #---loop through time..
    cmds.refresh(su=True)
    sampleFrames = [] #--generate sample frames...
    
    print "startF=%s endF=%s sampleRate=%s shape=%s" % (startF, startF+numSamples, sampleRate,shape)
    #print "range is ",range(startF, startF+numSamples, sampleRate)

    for i in timeRange:
        #print "time is ",i
        cmds.currentTime(i)
        for p in range(numPoints):
            vPos = cmds.pointPosition( shape+'.vtx['+str(p)+']',w=True)
            vFormat = "fff" #-- add a "less than" character at the begining of the string.. i removed it becuse my wordpress code syntax was doing goofy stuff"
            thisVertex = struct.pack(vFormat, vPos[0], vPos[1], vPos[2])
            file.write(thisVertex)

    
    cmds.refresh(su=False)
    file.flush()
    file.close()
    print filename," was sucessfully written out"
    return True
    
#example    
out = r'C:\testMaya\test.pc2'
shp = cmds.listRelatives(shapes=True)
writePC2File(out,shp[0],startF=1000,endF=1200,samplesPerFrame=2)