Over the last year, Google has approached the VFX community in a big way. Emboldened by the Zync purchase , the Google Cloud Platform has grown past the hurdles that made us apprehensive at first to become, in our opinion, the best solution for cloud rendering. At PipelineFX, Qube users across the world are starting to move in this direction, which means we have been getting more questions about it - namely, how do I get started and how can I make this easier on myself?
Here are 9 tips that will help blend render farm management with GCP today.
1. Pick Your Zone
Knowing your zone can help you control how fast your data is transferred to and from the cloud, how much you pay for transfers, which equipment may be available to you, and even your environment's stability.
Your data transfers more quickly when it's closer to you just like driving your car from one spot to another. Since all cloud providers offer their resources by geographic region, you should start by looking at the region you are in or that is closest to you. Within Google regions, they offer zones, and you can select zones within a region that may be physically closer to you. The closer you can get, the faster your data will transfer and the less it will cost to complete those transfers.
And if you have to transfer data from one cloud resource to another, know that transferring between zones in a region is cheaper than going across regions.
Learn about the zones available to you as it can affect the hardware and services available to you. Certain zones have Ivy Bridge or Haswell processors where others don't. In addition, zones may have different quotas on the available services like firewalls, IPs, image storage, etc.. Last but not least, for maximum uptime, Google highly recommends having instances spread out to several zones in the same region because zone maintenance and outages should be expected.
2. Pick Your Machine Type
Different machine types can offer a varying amount of processors or RAM which may render different applications more or less efficiently. It's best to do a set of test renders of your typical work on several different types to find the sweet spot in terms of price and performance.
For scale, look at Instance Groups. You can define a template for your instance image, so adding more processors to your standard render farm image is as simple as increasing the number of instances for that template. You can even set the instances to scale up automatically as you need additional resources. Instance Groups automatically load balance between the resources in the group, and if an instance crashes, it will automatically reboot and pick up where it left off.
3. Leverage Pre-emptable Instances
If you are scheduling render jobs that can afford the possibility of being paused or stopped for a period of time then Preemptible VM instances are a great bargain, running at a much lower price. By using these instances, you can save, on average, between 3050% of the hourly cost for rendering instances. You can save quite a bit of money across one or more projects, but with some added risk since there is no guarantee that the instance will be not be preempted.
4. Preview Jobs First
Optimize your use of cloud nodes. If you are going to be paying to use off site resources, then make sure the jobs are correct before you start the meter.
If you are using our render management solution, Qube!, you can leverage our Preview Frames feature to automatically split a job into two parts: the first test renders a sample set of frames immediately and the second is the remaining frames are dispatched already blocked, waiting on user approval to continue. The Preview Frames are normally rendered locally with the remaining frames destined for the Cloud.
This way, you don't start the meter on throw away work, and as you get information about the completed preview frames (render times, memory footprint, etc ), you can better determine how many cloud instances will be required to finish the project on time.
5. Don't Keep the Meter Running
A best practice in minimizing cloud costs is to pause or stop the instances when they are not in use. It can mean the difference between a large monthly bill versus the kind of cost you were expecting.
When rendering, it's best to automate. For example, Qube! offers the use of pre- and post-flight scripts that can be run at the start and end of every job. Write a script that starts your instance to run every single time a new job is dispatched to a cloud resource, and do the same to shut them down or pause them when your job completes. Your instances will only be up for rendering, and you will not be paying for cloud resources that are sitting idle.
6. Let the Network Expert Get You Connected
Connecting your network to the Cloud requires some technical expertise that's best handled by network engineers or system administrators. It's not that it's especially difficult, it's just that it takes some time and testing. It's getting better now that Google has added a more out of the
box VPN product and there is a growing online knowledge base where users are sharing their experiences.
7. Cache is King
In any network environment, whether it's on-premise or remote or a mixture of the two, the quality of experience and overall throughput depends on the strength and speed of your available cache. It makes an even bigger difference when you are using cloud nodes.
Avere's vXFT is an extremely easy way to initiate the transfer of data in and out of the Google Cloud.
Mount the Avere cluster along your file path, and your files will start transferring without changes to your pipeline. A standard three-node cluster can serve well over 100 cloud nodes, and the proprietary algorithms insure the most used files are available as the priori
