By Connor Lee, Incoming NYU Computer Science Student
Image created by editor with Midjourney
With the leap in AI progress making shockwaves throughout mainstream media since November 2022, many speculate their jobs will be taken over by their AI counterparts. One profession, however, cannot be possibly replaced: the researchers advancing deep neural networks and other machine learning models – the humans behind the AI. Although research is traditionally done within university walls, AI is by no means a traditional research field. A sizable portion of AI research is done in industrial labs. But which sector should aspiring researchers flock toward? Academia or industry?
“Academia is more inclined to basic fundamental research while the industry is inclined to user-oriented research driven by the large data access,” says Nitesh Chawla, a Professor of Computer Science and Engineering at the University of Notre Dame. Prof. Chawla points to the pursuit of knowledge as a separating factor between industrial and academic AI research. Within the industry, research is tied to a product, advancing towards a better society-- while within academia, the pursuit of pure discovery drives research breakthroughs. The seemingly endless academic freedom does not come without its drawbacks, “academia does not have the data nor the computing access available,” according to Prof. Chawla.
For aspiring young researchers, the choice seems simple: the private sector has everything they could want. Vast, autonomous, commercial organizations striving toward innovation while supported by readily available data, computing power, and funding. This led to a perception that the industry is “stealing” talent away from academia. Academics, naturally, complain. A study published in 2021 by a team from Aalborg University pointed out that “increasing participation of the private sector in AI research has been accompanied by a growing flow of researchers from academia into industry, and especially into technology companies such as Google, Microsoft, and Facebook”.
As expected, industrial researchers disagree. “When I hire for my team, I want top talent, and as such I’m not poaching academic talent, but rather I am trying to help them get industry awards, funding from industry, and have their students as interns,” explains Dr. Luna Dong, a Principal Scientist at Meta who is the head scientist working on Meta’s smart glasses. She sees a glaring difference between industry and academia, which could be credited to the fundamental way research is conducted. According to Dr. Dong, AI research within an industry is conducted by knowing what the end product should look like and reverse engineering a path toward it. In contrast, academics, having a promising idea, continuously construct various paths, not knowing where those paths would lead.
Yet, despite these contrasts, Dr. Dong believes the industry helps academia and vice versa, “lots of industry breakthroughs are inspired by applying the research from academia on real use-cases.” Likewise, Computer Science Professor Ankur Teredesai from the University of Washington, Tacoma, describes the relationship between industry and academia as supporting each other, “symbiotic is the word that comes to mind.” As he views it, research practices have evolved into academics shifting their agenda to aid industry products -- a good example of that shift would be joint positions within major corporations that some prominent professors are holding.
Regardless of their affiliations, the data science community converges together a few times a year at conferences. Prof. Chawla describes them as a “wonderful melting pot.” Some conferences are traditionally more academic, some purely industrial but some are a perfect blend of both. Prof. Chawla points to KDD, or the Special Interest Group on Knowledge Discovery and Data Mining, a conference known for such a connection. KDD maintains two parallel peer-reviewed tracks: the research track and the applied data science (ADS) track. As put by Dr. Dong, who was the ADS Program Co-Chair at KDD-2022, “KDD is helpful by providing a forum for researchers and practitioners to come together to listen to the talks and discuss the techniques while inspiring each other. KDD is a place where we break the barriers of communication and collaboration, where we demonstrate how data science and machine learning advances with industry consumption.”
This is the mindset that drove KDD from its early days. “One of the things we wanted to do from the very beginning was to create a conference where applications were well represented,” commends Prof. Usama Fayyad, Executive Director of the Institute for Experiential AI at Northeastern University and a former Chief Data Officer of Yahoo, who together with Dr. Gregory Piatetsky-Shapiro co-founded the KDD conference in 1995. Prof. Fayyad believes that if AI conferences were only focused on academics, it would be a big miss due to the collective desire to prove research on real problems and motivation to drive new research based on emerging data sets.
However, opening up KDD to the industry also had its challenges. With the research track being rightfully dominated by academia-originated work, the ADS track should have been primarily dedicated to applied studies coming from industrial research labs. In reality, more than half of ADS publications have their origins within academia or are a result of strong academic-industrial collaboration. A decade ago, Prof. Fayyad realized that many interesting AI applications were developed by teams that were simply too busy to write papers. He led KDD into its current phase, where KDD organizers venture and curate distinguished invited talks given by top industrial practitioners. The ADS invited talks have quickly become the highlight of the conference.
The KDD Cup competition held annually in conjunction with the KDD conference, is yet another way to connect the academic and industrial worlds. “KDD Cup is a way to attract both industry and academia participants where companies bring some of the challenges that they are comfortable sharing, while academics get to work on data they would never have access to,” describes Prof. Teredesai, who is also the CEO of a health tech company CueZen. Each year, a novel task is introduced and a new dataset is released. Hundreds of teams sprint towards the most effective solution, competing for prizes and fame. Prof. Fayyad agrees, “It's been a very healthy thing for the field because we see participation from academia, students diving in, or even companies teaming together”.
Circling back to the choice between industry and academia, it will soon become irrelevant. With academic courses taught by practitioners, professors leading industrial labs, global cloud computing resources becoming dominant, and more data becoming available, the academic-industrial boundaries are quickly getting blurred in the AI domain. No need to stick to any of the two sectors, just choose the project you are most excited about!
Connor Lee is a 2023 graduate from Saratoga High School in the Bay Area. He will be joining the Computer Science program at NYU in the fall. By all means, Connor will be one of the youngest KDD attendees ever!