The new Snowpark Containers release is now generally available — this is super exciting as it means you can now prototype on top of Snowflake more rapidly than ever before.

Snowpark Containers are a relatively new feature from our friends at Snowflake. When it launched last year, it introduced the capability for running containerised serverless Services and Jobs directly within Snowflake. Essentially, this means that Snowflake’s compute power can now execute code from Docker containers—a familiar tool for many engineers.

We love this sort of stuff at Tasman. Throughout the year, we organise Hackdays to explore new tools & tech; and in March 2024, our Data Engineering Team took a deep dive into this Snowflake feature with the aim of running a containerised dbt project within a Snowpark Container.

However, that initial excitement was somewhat dampened by certain limitations. At the time, Snowflake had only released Snowpark Container Services, which are intended to be always-on—similar to an API or a web service.

Although we succeeded in running a dbt job, the system perceived the job’s completion as an error and automatically restarted it — leading to a continuous loop that consumed resources unnecessarily. (Oops!) What we actually needed was a Container Job, a workload that terminates upon completion, which unfortunately wasn’t available then.

Fast forward to July 2024 — when Snowflake launched Snowpark Container Jobs in Preview Mode. We revisited our Hackday project and, with a few adjustments, successfully executed the dbt project without the previous issues! 🎉

This experience opened our eyes to the myriad possibilities that Snowpark Containers offer. These containers can handle a variety of data tasks—everything from data ingestion and transformation processes to sending reports or even reverse ETL tasks that integrate enriched data back into tools like Salesforce. This is all achievable within Snowflake’s serverless architecture, eliminating the need for external cloud providers such as GCP or AWS.

As we continue to explore Snowpark Containers, we are increasingly optimistic about their potential. We look forward to sharing more insights as we further integrate this technology into our operations. Stay tuned!