Startup culture vs Research culture
Having worked with tech startups who have a strong academic/ research background and at startups that even had specific research teams at a very early stage, I started reflecting on the difference between research culture and startup culture. There’s no idea to beat around the bush: they’re very different. Based on my experience, there are more differences between the two than common denominators for a harmonious coexistence.
There are some startups, such as Deepmind and OpenAI, that have had research as an absolute core part of their organization from the getgo and have become successful in their own ways (e.g. getting acquired by Google or pivoting towards a more commercial framework by offering massive AI models such as GPT-3 behind paid APIs). However, trying to emulate these is not a good idea if you’re a startup pitching to investors. Companies usually have to first achieve wild commercial success to later have the resources and capabilities (read: luxury) to invest in teams that are solely dedicated to research — not vice versa (excluding specific industries such as pharma).
At any startup that I’ve heard of with a more or less dedicated research arm, there’s been a smaller or larger shock in regards to how research culture is different from the general startup culture. This usually starts from what the company values to how you set the goals, measure progress, motivate people, etc. Research is a very different world compared to the messy startup world. An example of something that was hard for me to understand when working with a startup with a research arm was how important publications are. I personally didn’t then have a mental framework for it (coming from the business side) and it took me a while to understand — now I think I somewhat get it. The challenge in a startup is how you’re going to get that to coexist with teams and people that are going to measure themselves based on e.g. a product growth goal with people who measure themselves based on the number of citations they get. It might sound to an outsider that those might be complementary, but in reality, it turns out that you have to do a lot more work than you think to get those to coexist.
Getting a startup to exist happily in the middle of even three different organizational cultures (e.g. commercial, engineering, product) is usually already a challenge. I can guarantee you that it will become a lot more challenging by adding a fourth dimension (research), which is in most cases motivated by completely different parameters than the three mentioned before.
I think the starkest difference between research culture and startup culture can be found in the AI/ML community. As ML usage in the real world is still fairly new, most people with ML expertise have gained it through academia: taking courses, doing research, reading papers. Hence they often experience a culture shock when they deploy their first ML system into the wild. ML in production is very different from ML in research. Below are five of the major differences from Stanford’s Machine Learning Systems Design course.
In the ML community, research and leaderboard projects often have one single objective: model performance (e.g. developing a model that achieves state-of-the-art results on benchmark datasets). To get small improvements in performance, researchers often utilize techniques that make models too complex to be truly useful in an operational setting for a startup.
Things that go into production in a startup (or company of any size for that matter) having different objectives from research is one of the reasons why successful ML research projects are often not used in production. As a concrete example, ensembling is a technique popular among the winners of many ML competitions, including the famed $1M Netflix Prize. It combines “multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone.” While it can give you a small improvement, ensembled systems risk being too complex to be useful, e.g. more error-prone to deploy, slower to serve, or harder to interpret.
For many tasks, a small improvement in performance can result in a huge boost in revenue or cost-saving. For example, a 0.2% more efficient energy use can result in millions of dollars saved for companies like Google. However, for many tasks, a small improvement might not be noticeable for users. From a user’s point of view, a speech recognition app with a 94.5% accuracy is not that different from an app with a 94.7% accuracy. For the second type of tasks, if a simple model can do a reasonable job, complex models must perform significantly better to really justify the complexity. As a startup, you have to place an emphasis on cutting through the hype and building things that actually work and deliver real business value — not focus on what’s state-of-the-art from a research perspective.
We can see how more and more models are being commoditized, resulting in that you can as s startup founder just use off-the-shelf models. The true bottleneck in the majority of startups is data and real-world data is nowhere close to research data. The real problem is how do you collect data and verify data, how do you cope with constant distribution shifts and data drift? Not how state-of-the-art your model is.
I believe that the toughest clash between research culture and startup culture comes from the power-law nature of building a successful startup that will eventually become a big successful company (which could employ a full-time research team). You’re especially in the beginning of a startup’s life-cycle constantly living on a knife-edge. To quote Andreessen Horowitz:
“There is an underappreciated aspect of success to consider: tough moments make for the most creative solutions. If you have the cushy safety net of your academic/research position, there is an option to fail, and given how demanding startups are, there are going to be do or die moments that require everything you’ve got. You have to be willing to give that.”
All this being said — there are many successful startups that have been built at the back-heel of research; a prime example being the current Silicon Valley data darling Databricks. But there the whole team went all-in, placing all their eggs in one basket to build a commercial software company, not to do research. There’s no other way.
Hence, if not absolutely necessary due to the nature of the business, don’t include researchers in your tech startups org chart (this does absolutely not mean that you shouldn’t have people in your org chart that have done research in the past). It might sound cool in your pitch deck to have an internal “Deepmind” — but for many investors, this will be a big red flag. Not because it increases your burn rate and decreases your runway (researchers aren’t cheap), but mainly because the cultural balance and incentives are extremely challenging to get right.
Recommended further readings:
📚 Power Law for Professors: Why you should put all of your eggs in one basket by Andreessen Horowitz
📚 Researchers and Founders by Sam Altman (CEO of OpenAI and the former president of Y Combinator)
