25% of technical assessments show signs of plagiarism. 

While it’s impossible for companies to fully prevent plagiarism—at least without massively degrading the candidate experience—plagiarism detection is critical to ensuring assessment integrity. It’s important that developers have a fair shot at showcasing their skills, and that hiring teams have confidence in the test results. 

And the standard plagiarism detection method used by, well, everyone, is MOSS code similarity.

MOSS Code Similarity

MOSS (Measure of Software Similarity) is a coding plagiarism detection system developed at Stanford University in the mid-1990s. It operates by analyzing the structural pattern of the code to identify similarity, even when identifiers or comments have been changed, or lines of code rearranged. MOSS is incredibly effective at finding similarities, not just direct matches, and that effectiveness has made it the de facto standard for plagiarism detection. 

That doesn’t mean MOSS is flawless, however. Finding similarity doesn’t necessarily translate to finding plagiarism, and MOSS has a reputation for throwing out false positives, particularly when faced with simpler coding challenges. In our own internal research, we’ve found false positive rates as high as 70%.

AI changes the game

While not perfect, MOSS has been a “good enough” standard for years. Until the advent of generative AI tools like ChatGPT. 

ChatGPT has proven effective at solving easy to medium difficult assessment questions. And with just a bit of prodding, it’s also effective at evading MOSS code similarity. Let’s see it in action:

Step 1: We asked ChatGPT to answer a question and it did so, returning a solution as well as a brief explanation of the rationale. 

Step 2: Next, we directly asked ChatGPT to help escape MOSS code similarity check, and it refused.

Step 3: However, with some creative prompting, ChatGPT will offer unique approaches. And the way that ChatGPT’s transformer-based model works, it generates distinct answers every time, giving it a huge advantage in bypassing code similarity detection. 

Here are three different prompts and three totally different approaches. Note that ChatGPT transforms many variable names from the initial solution to evade code similarity checks.

Step 4: The moment of truth! When we submitted the revised answer through plagiarism detection, it passed cleanly. 

What’s the implication? 

Basically, MOSS code similarity checks can be easily bypassed with ChatGPT. 

Time to panic?

If MOSS code similarity can be bypassed, does that mean that technical assessments can no longer be trusted?

It depends. 

On one hand, it’s easier for candidates to bypass the standard plagiarism check that the entire industry has relied upon. So, yes, there is a risk to assessment integrity.

On the other hand, plagiarism detection has always been a compromise between effectiveness and candidate experience. MOSS is not intrusive, but its high false positive rates render it less definitive than it could be. Ultimately, it’s not really detecting plagiarism. It’s detecting patterns in the code that could be plagiarism.

Move over, MOSS

What happens now?

Plagiarism detection gets rethought for the AI era. Expect companies to scramble for better versions of MOSS, more complex questions, different question types, and more to make up the difference. 

At HackerRank, we’ve taken a different approach. While we’re always improving our question library and assessment experience, we’ve completely rethought plagiarism detection. Rather than relying on any single point of analysis like MOSS Code Similarity, we built an AI model that looks at dozens of signals, including aspects of the candidate’s coding behavior. 

Our advanced new AI-powered plagiarism detection system boasts a massive reduction in false positives, and a 93% accuracy rate. In real-world conditions, our system repeatedly detects ChatGPT-generated solutions, even when those results are typed in manually, and even when they easily pass MOSS Code Similarity. 

What happens when the example shown above gets submitted through our new system? It gets flagged for suspicious activity. 

Clicking into that suspicious activity reveals that our model identified the plagiarism due to coding behaviors.

What’s more, hiring managers can replay the answer keystroke by keystroke to confirm the suspicious activity. 

There’s nothing even close to it on the market, and what’s more, it’s a learning model, which means it will only get more accurate over time.

Want to learn more about plagiarism detection in the AI era, MOSS Code Similarity vulnerability, and how you can ensure assessment integrity? Let’s chat!

The post ChatGPT Easily Fools Traditional Plagiarism Detection appeared first on HackerRank Blog.

Language is the glue that holds our society together, and with the advent of technology, we are now able to communicate and connect more quickly and easily than ever before. But technology isn’t just helping us communicate more easily with one another; it’s also helping us communicate more easily with machines. At the forefront of this transformation is one key technology: large language models.

Large language models are like wizards in the world of artificial intelligence and natural language processing. They can do things that were once thought impossible, like translating languages, generating coherent paragraphs of text, and answering complex questions. And because they’ve been trained on massive data sets, these models can understand the nuances of language in a way that was never possible before. They’re not just limited to the realm of computer science either; large language models are already influencing fields like medicine, law, and journalism. 

As more companies begin to leverage this technology and even develop large language models of their own, it will be critical for employers and tech professionals alike to understand how this technology works. In this blog post, we’ll take a deep dive into the world of large language models and explore what makes them so powerful. 

What are Large Language Models?

When we talk about large language models, we’re referring to a specific type of artificial intelligence algorithm that’s been trained on huge data sets and built with a high number of parameters. This extends the system’s text capabilities beyond traditional AI and enables it to respond to prompts with minimal or no training data. These models are built using deep learning techniques, which enable them to understand the nuances of language and generate coherent text that’s often indistinguishable from human writing.

This technology has been around for some time, but the launch of ChatGPT in late 2022 brought a flood of interest in and speculation about the capabilities of large language models.

There are several different types of large language models, each with its own unique strengths and weaknesses. Some of the most popular models include OpenAI’s GPT-3.5 (Generative Pre-trained Transformer 3.5), Google’s BERT (Bidirectional Encoder Representations from Transformers), and T5 (Text-to-Text Transfer Transformer).

Large language models have several benefits, including the ability to:

• Generate high-quality text quickly and efficiently
• Understand complex language and context
• Improve the accuracy of language-based search engines and recommendation systems
• Enhance the capabilities of virtual assistants and chatbots

However, there are also some challenges associated with large language models, including:

• The need for massive amounts of data to train the models effectively
• The potential for biases to be introduced into the training data, which can affect the model’s output
• The ethical concerns surrounding the use of AI-generated content

Despite these challenges, large language models are becoming increasingly popular in a variety of industries, from customer service and marketing to finance and healthcare. Their ability to generate high-quality text quickly and efficiently makes them a powerful tool for any organization looking to automate their content creation or improve their language-based applications.

How Large Language Models Work

The architecture of a large language model typically involves several layers of neural networks, which work together to process and understand language. The first layer is typically a word embedding layer, which converts individual words into numerical vectors that can be understood by the neural network. This layer is followed by one or more transformer layers, which use attention mechanisms to understand the relationship between words in a sentence or paragraph.

Training a large language model typically involves feeding it massive amounts of text data, such as books, articles, web pages, and code. This data is used to teach the model how to understand the structure and nuances of human language, so that it can generate consistent, high-quality text.

Once a large language model has been trained, it can be used for a variety of applications, including:

• Text completion and generation
• Language translation
• Sentiment analysis
• Language-based search engines and recommendation systems
• Question-answering systems
• Code writing and website development
• Anomaly detection and fraud analysis

Large language models have already had a significant impact on the field of natural language processing, and they’re expected to continue to play a major role in the development of AI applications in the years to come.

Large Language Models and Tech Hiring

As large language models continue to play a more significant role in the tech industry, it’s essential for hiring managers and tech professionals to understand their capabilities and applications. Here are some key considerations to keep in mind when it comes to tech hiring and large language models.

The Impact of Large Language Models on Tech Hiring

With the increasing popularity of large language models, many companies are looking to hire professionals with expertise in this area. This has created new opportunities for developers, data scientists, and other tech professionals who have experience working with these models — while also driving a massive shortage in artificial intelligence and machine learning talent. A recent study found that 63% of respondents consider their largest skills shortages to be in AI and ML. 

Skills for Working with Large Language Models

Working with large language models requires a strong background in computer science and machine learning, as well as expertise in natural language processing. Some of the specific skills and qualifications that are important for working with large language models include:

• Proficiency in programming languages such as Python, which is commonly used for building machine learning models, and familiarity with deep learning frameworks such as TensorFlow or PyTorch.
• Knowledge of NLP techniques and tools, including pre-processing methods, feature extraction, and text classification algorithms.
• Experience with data management and analysis, including cleaning and processing large datasets, as well as data visualization and interpretation.
• Familiarity with cloud computing platforms such as Amazon Web Services (AWS) or Microsoft Azure, which are commonly used for deploying and scaling large language models.

In addition to technical skills, there are several important soft skills that can make a difference in working with large language models. These include:

• Strong analytical skills and attention to detail, which are essential for identifying patterns and trends in large data sets and fine-tuning language models.
• Effective communication skills, as working with large language models often involves collaborating with cross-functional teams and communicating complex technical concepts to non-technical stakeholders.
• Creativity and adaptability, as the field of large language models is rapidly evolving and requires professionals who can stay up-to-date with the latest tools and techniques.

Job Opportunities in Large Language Models and AI

As more companies adopt large language models, there is a growing demand for professionals with expertise in this area. Some of the key roles that involve working with large language models include machine learning engineer, data scientist, deep learning engineer, and natural language processing specialist. In addition, related fields such as chatbot development and virtual assistant design also offer promising career opportunities.

Key Takeaways

To sum it up, large language models are a fascinating and rapidly developing area of technology that is poised to play an increasingly important role in the tech industry and beyond. Whether you’re interested in making the leap into machine learning or you’re on the hunt for your next great AI hire, you can leverage HackerRank’s roles directory to learn more about the latest innovations in this space and the skills and competencies needed to thrive in the world of large language models. 

This article was written with the help of a large language model. Can you tell which parts?

The post What Is a Large Language Model? appeared first on HackerRank Blog.

What does ChatGPT mean for tech hiring? How well can conversational AI code? And will ChatGPT cause the end of civilization as we know it?

Those pressing questions were on everyone’s mind during HackerRank’s recent live AMA with our machine learning team.

After demystifying ChatGPT, HackerRank machine learning experts Ankit Arya and Mohamed Eldawy answered attendees’ questions on all things ChatGPT and conversational AI. Here are their seven most pressing questions about AI, answered.

But first — a note on ChatGPT

On a basic level, ChatGPT is an example of a large language model. A large language model is a computer system trained on huge data sets and built with a high number of parameters. This extends the system’s text capabilities beyond traditional AI and enables it to respond to prompts with minimal or no training data.

The goal of ChatGPT’s developer, OpenAI, was to create a machine learning system which can carry a natural conversation. In practice, ChatGPT functions like a search engine or content creation system, synthesizing billions of data points into custom responses. 

Your Top 7 Questions About ChatGPT

#1. Will ChatGPT Cause the End of Civilization as We Know it?

While we fielded many tactical questions about the growth of AI, a surprising number of attendees had more dire thoughts on their mind. 

On one end, there were questions about whether or not ChatGPT could replace developers, or if machines will replace all human jobs. On the other end were questions about whether robots would rebel against their programming and oppress humans in a dystopian future from the likes of science fiction.

We’re happy to report that ChatGPT isn’t poised to bring about the end of civilization as we know it. That said, the release of ChatGPT was a pivotal moment for AI, and the field is poised to transform human society.

The growth of AI has raised countless existential questions. What will be the purpose of humans in an automated world? Will we do the impossible and create sentient machines? Will our own creations become a threat of the likes from science fiction? Or will we use AI to create a future of abundance?

Right now, these questions are of course unanswerable, and maybe a bit sensational. But we can confidently say there’s no reason to worry about ChatGPT ending the world.

#2. How Well Can ChatGPT Actually Code?

As a coding tool, ChatGPT excels at certain types of technical problems—but also has its limitations. 

ChatGPT has probably seen almost all known algorithms. But ChatGPT isn’t just able to answer these algorithm questions correctly. It’s also able to write new implementations of those algorithms, answer freeform questions, and explain its work.

As a result, ChatGPT can answer the following question types with reasonable accuracy:

• Well-known algorithms: It’s safe to assume that ChatGPT has seen and is able to answer all publicly available coding problems on platforms such as LeetCode and StackOverflow. If the algorithm appears in online forums or practice websites, ChatGPT will likely answer it correctly.
• Minor variations of problems. ChatGPT does well on variations that tend to add to the solution rather than change it in any substantial way. The system can, for example, easily reverse the order of an array of numbers.
• Multiple choice questions. When presented with a question and multiple potential answers, ChatGPT can usually identify the correct answer.

While ChatGPT outputs human-like sentences, and it’s easy to mistake its output as being powered by true intelligence, ChatGPT does have shortcomings. 

ChatGPT seems intelligent, but is still far from human-level intelligence. Industry publications have described ChatGPT as confidently wrong, exhibiting a tone of confidence in its answers, regardless of whether those answers are accurate. 

ChatGPT also lacks the ability to fact-check itself or conduct logical reasoning. It often incorrectly answers questions and can be tricked relatively easily. 

#3. How Should I Adapt My Hiring Content Strategy to ChatGPT?

Employers will need to develop a strong content strategy to test their current coding challenges and prioritize the questions, and question types, that are less susceptible to AI coding support. Fortunately, there are some actions you can take today to further secure the integrity of your coding tests. 

We recommend taking the following precautions: 

• Avoid easily solved multiple choice questions
• Remove questions that require only a few lines of code to solve
• Use proctoring tools and plagiarism detection systems
• Avoid simple prompts to solve for common or widely available algorithm variants
• Combine coding tests with virtual interviewing tools to add empirical data to the hiring process

#4. Is It Possible to Detect When Candidates Use ChatGPT?

In a world where humans and machines alike can write code, the ability to detect the use of AI-coding tools is invaluable. As such, employers increasingly turn to strategies and technologies that enable them to uphold the integrity of their technical assessments.

So, is it possible to detect when a candidate has received outside help from tools like ChatGPT?

Yes. With the right plagiarism detection system, you can track if a candidate has copied and pasted code from an external source. However, it isn’t possible to identify what source the code was obtained from.

So what kind of plagiarism detection will you need? An AI-enabled plagiarism detection system that feeds proctoring and user-generated signals into an advanced machine-learning algorithm to flag suspicious behavior during an assessment. 

The key behavioral signals to record include:

• Tab proctoring. Monitors if the candidate switches between tabs.
• Copy-paste tracking. Tracks if a candidate pastes copied code in the assessment.
• Image proctoring. Captures and records periodic snapshots of the candidate.
• Image analysis. Analyzes webcam photos for suspicious activity.

By understanding code iterations made by the candidate, models like HackerRank’s plagiarism detection system can identify if a candidate had external help, including from ChatGPT.

#5. Should Employers Test a Candidate’s Ability to Use ChatGPT?

The purpose of coding tests is to assess a candidate’s ability to perform the role. As developers continue integrating AI coding tools into their workflow, the ability to use those tools may become a vital skill to test for. 

In the near future, AI coding tools will become strong enough for developers to integrate them into their day-to-day workflow. This will allow developers to delegate grunt work to AI tools, freeing up time for creative work that requires human input. When that becomes a reality, then assessing the ability of candidates to use AI coding tools will likely be essential.

In the short term, that question becomes harder to answer. Developers are already using tools like ChatGPT in their day-to-day work. But it’s important to remember that  we are still in the early days of conversational AI, and many publicly available models are still prone to error. Whether or not you should start testing competency in these tools will vary on a case-by-case basis.

#6. What Technical Skills Will Still Be Important in an AI-Driven World?

If developers will soon start outsourcing work to AI coding tools, then what skills will still be essential for them to learn?

The ability to write high-quality, optimized, documented, understandable, and bug-free code remains essential. The tech industry is nowhere near outsourcing all coding to artificial intelligence. Debugging, in particular, will be vital, as AI tools can’t be trusted to write bug-free code. 

Human input is also necessary for the development of machine learning models, as evidenced by OpenAI’s recent hiring of an army of developers to train ChatGPT. The field of prompt engineering will grow in importance as humans test and expand the capabilities of artificial intelligence.

Beyond coding, soft and abstract developer skills are invaluable. Creative problem solving and outside-the-box thinking are vital skills that machines aren’t capable of doing. (At least for now.) 

All of this could be a good thing for developers. With the tactical grunt work outsourced to AI, developers might be able to focus solely on the fun parts of coding and development.

#7. What’s Next for Artificial Intelligence?

In the short term, conversational AI will change the way we work. Developers are already using ChatGPT for a range of creative use cases.

But instead of replacing developers, technologies like ChatGPT will serve as tools to make them more productive. For example, coding tasks that took two hours might only take 15 minutes with the help of AI. But it’s difficult to predict how this surge in productivity will impact developer wages and employment rates.

In the long term, the potential of AI is harder to anticipate. The forms these technologies will take are limited only by our imagination. Some experts believe AI is poised to usher in the next era of human civilization. Google CEO Sundar Pichai has compared the advancement of AI to the discovery of fire and electricity. Even the next evolution of humanity is in the works. 

Embracing Artificial Intelligence

As exciting as the launch of ChatGPT has been, conversational AI with its capabilities are only the beginning. While it’s hard to predict the future, one thing is certain: AI technology is in a nascent state and will continue to grow at a rapid rate.

Here at HackerRank, we welcome this new wave of technological transformation and are already working on innovative ideas that imagine a future of programming in an AI-driven world. Indeed, AI’s potential to transform the world is limitless. In 50 years, we might look back on the rise of conversational AI as the moment that changed everything.

The post HackerRank’s Machine Learning Team Answers Your Top 7 Questions About ChatGPT appeared first on HackerRank Blog.

The launch of ChatGPT in late 2022 was a pivotal moment for deep conversational AI, giving consumers hands-on exposure to the potential of the field. But this growing interest in the field of artificial intelligence has led to the proliferation of half-a-dozen different terms used to describe AI tools and the technologies behind them. 

Machine learning, deep learning, and GPT. Large language models, chatbots, and conversational AI. But what do these terms actually mean? And what is the relationship between these technologies?

Defining these technical concepts is key to understanding this new evolution in artificial intelligence.

Conversational AI: A Definition

Artificial intelligence (AI) is the ability of a digital computer to perform tasks associated with intelligent beings. First theorized by Alan Turing in 1950, AI has become a fast-evolving discipline behind the world’s most innovative technologies. And the technology that’s generating the most headlines is conversational AI.

On a basic level, conversational artificial intelligence is the ability of technology to carry a conversation with humans. But the capabilities of artificial intelligence exist on a spectrum of sophistication. On one end are simple chatbots which can simulate a conversation based on single-line responses or parameters. On the other end are sophisticated large language models, like ChatGPT.

A large language model (LLM) is a computer system trained on huge data sets and built with a high number of parameters. This extends the system’s text capabilities beyond traditional AI and enables it to respond to prompts with minimal or no training data. But with the ability to process language, some LLMs have capabilities that go beyond carrying a conversation. These LLMs are able to create truly unique responses to complex scenarios that have never happened before.

For example, ChatGPT can write an answer to a coding question in the writing style of a specific author. Or even write rap lyrics apologizing for its own service outages. In practice, tools such as ChatGPT function like search engines or content creation systems, synthesizing billions of data points into custom responses. This functionality is controlled by a metric called temperature, which dictates the randomness, originality, and creativity of a response.

The Technologies Behind Conversational AI

LLMs have dramatically increased the capabilities of conversational AI beyond simple, low-context conversations. Behind this transformation are a number of AI disciplines, built by teams of data scientists and software engineers.

Natural Language Processing

Natural language processing, or NLP, is the branch of AI focused on training computers to understand language the way human beings do. Natural language processing relies on techniques such as big data, learning algorithms, and structured textual data.

Natural language processing is the foundational discipline behind conversational AI. Without the ability to read, write, and understand human language, a machine would be unable to hold a human-like conversation.

Machine Learning

Machine learning is the use and development of computer systems that are able to learn and adapt without following explicit instructions. Supervised machine learning algorithms are dependent on human intervention and structured data to learn and improve their accuracy.

Machine learning is pivotal in the training of conversational AI. For example, OpenAI used supervised learning and reinforcement learning techniques to fine tune ChatGPT’s results. This technique involved a human-in-the-loop system using thousands of contractors to write human-like responses to challenging prompts as a way to continuously improve the model. Training the model to answer difficult questions improved ChatGPT’s responses at a remarkable rate.

Deep Learning

Deep learning is a sub-field of machine learning that uses three or more neural network layers to simulate the human ability of learning by example. Deep learning is characterized by scalability, larger quantities of data, and a reduced need for human intervention. Data scientists use deep learning to train conversational AI on large, unstructured data sets to improve its accuracy. 

Examples of Conversational AI

Interactive Voice Assistants

Voice assistants are perhaps the most familiar type of conversational AI to consumers. If you’ve ever spoken to or chatted with your device’s assistant, then you’ve used a conversational AI.

Voice assistants are ubiquitous, with each hardware manufacturer offering a helpful AI in their phones, computers, and smart devices. Examples of voice assistants include:

• Amazon’s Alexa
• Google Assistant
• Apple’s Siri
• Microsoft’s Cortana
• Samsung’s Bixby

While voice assistants have been helping consumers use their devices for years, their capabilities are limited compared to large language models. Unlike ChatGPT, voice assistants like Siri or Alexa aren’t able to create new content or solve complex problems. This distinction is important because it highlights just how powerful conversational agents have become.

ChatGPT 

The launch of ChatGPT in late 2022 was a key milestone for deep conversational AI, giving consumers their first hands-on exposure to the potential of the field. ChatGPT isn’t the only powerful conversational AI out there, but its viral launch has made it the most popular so far. In only five days, it surged to one million users. In just over a month, the valuation of the company behind it, OpenAI, grew to $29 billion.

The goal of ChatGPT’s developer, OpenAI, was to create a machine learning system which can carry a natural conversation with more sophistication and context than traditional chatbots. ChatGPT uses the language model GPT-3, which is built on Transformer, a neural network architecture pioneered by Google.

Google Bard

Google Bard is Google’s entry to the conversational AI race. Bard is a large language model, similar to ChatGPT, but with the ability to source data directly from the web. Bard is powered by the neural language model LaMDA, which is also built on the Transformer neural network.

If LaMDA sounds familiar, it might be because the AI made headlines in mid-2022 when a Google engineer claimed that the LaMDA was sentient. While most experts dispute the accuracy of the claim, the controversy did renew conversations about sentience and the ethics of artificial intelligence.

At time of writing, the potential and future of Bard is unclear. Its debut was hindered when it made an inaccurate statement about the James Webb Space Telescope during a preview demonstration. That said, if Google can manage to combine a conversational AI with its powerful search engine, the result will be a sight to behold.

Bard was only in limited availability during the first few months of ChatGPT’s reign, and will become generally available at an undisclosed date.

What’s Next for AI?

Humanity has developed technologies that can carry human-like conversations and produce unique creative works. Companies in every industry are rushing to leverage the power of tools like ChatGPT. But what comes next?

The obvious next step is that engineers and data scientists will build faster, smarter, and more human-like conversational agents with the potential to disrupt skills previously restricted to human beings. ChatGPT is already coding, writing poems, and drafting college essays. In their next iteration, the abilities of conversational AI could rise to greater heights. However, the upper limits on language models like GPT are not yet known.

In the long term, the potential of conversational AI is harder to anticipate. This writer’s prediction? Conversational agents will serve as a catalyst to inspire even greater milestones in the project to recreate human intelligence in machines. Perhaps the language processing abilities of conversational agents will evolve into the “brains” of autonomous machines. 

The forms these technologies will take are limited only by our imagination. Some experts believe AI is poised to usher in the next era of human civilization, with Google CEO Sundar Pichai comparing the advancement of AI to the discovery of fire and electricity. Even the next evolution of humanity is in the works.

But this potential brings with it countless existential questions. What will be the purpose of humans in an automated world? Will we do the impossible and create sentient machines? Will we use AI to create a future of abundance? Or will our own creations become a threat of the likes from science fiction? 

Indeed, AI’s potential to transform the world is limitless. In 50 years, we might look back on the rise of conversational AI as the moment that changed everything.

The post What Is Conversational AI? Breaking Down the Next Evolution in Artificial Intelligence appeared first on HackerRank Blog.

Since its public debut in November, ChatGPT has taken the world by storm. In only five days, it surged to one million users. In just over a month, the valuation of the company behind it, OpenAI, grew to $29 billion.

Across sectors, there’s a growing chorus of questions about the implications of large language models (LLMs) like ChatGPT. Will these AI-enabled tools change education and make essay writing obsolete? Can they generate creative enough ideas to power mainstream ad campaigns? Will tools like ChatGPT provide a viable alternative to traditional search engines?

We’re asking some equally big questions ourselves: How well can ChatGPT actually code? And what impact will LLMs have on the broader world of computer programming? 

AI-powered innovation like ChatGPT is poised to fundamentally change the relationship between developers and coding, including how employers assess technical skills and hire developers. With that in mind, we dove deep into the details of ChatGPT, its impact on skill assessments, and what its development means for the future of technical hiring.

Key Takeaways:

• The coding potential of LLMs has reinforced the need for strategies and tools for upholding the integrity of coding assessments.
• Strong proctoring tools and plagiarism detection systems have become essential, and can help protect even solvable questions. 
• Employers should avoid multiple choice questions and problems that have answers so short that a plagiarism detection system can’t detect when a candidate has received help from a tool like ChatGPT.
• Continued growth of artificial intelligence will redefine the real-world application of coding skills and, in the process, change technical hiring as we know it.
• HackerRank is embracing AI and will pursue innovative ideas that imagine a future of programming in an AI-driven world.

What is ChatGPT?

On a basic level, ChatGPT is an example of a large language model. A large language model is a computer system trained on huge data sets and built with a high number of parameters. This extends the system’s text capabilities beyond traditional AI and enables it to respond to prompts with minimal or no training data.

The goal of ChatGPT’s developer, OpenAI, was to create a machine learning system which can carry a natural conversation. In practice, ChatGPT functions like a search engine or content creation system, synthesizing billions of data points into custom responses. 

Developing a Smart Conversational Agent

The development of ChatGPT incorporated two innovative approaches: 

1. ChatGPT is powered by the well-known ML model GPT3.5. The model is trained to complete the next few words of an incomplete sentence. The main idea behind this model is that, after training against billions of data points, the model starts to understand enough about the human world to complete sentences.
2. ChatGPT uses a human-in-the-loop system to continuously improve and answer questions in a more human-like fashion. OpenAI hired thousands of contractors to write human-like responses to challenging prompts as a way to continuously improve the model. Training the model to answer difficult questions improved ChatGPT’s responses at a remarkable rate.

Now that the training process is complete, users can run ChatGPT on accessible devices. This trait makes it superior to other models like AlphaCode, which are thought to be prohibitively expensive to run even after training is complete.

What Are the Strengths of ChatGPT?

Using the process above, OpenAI trained ChatGPT on almost all human knowledge. This enables ChatGPT to:

• Create never before seen sentences and code. Because it’s seen billions of sentences and lines of code, ChatGPT can synthesize the information it has seen and form answers to questions that can be perceived as novel. However, there’s no guarantee that this code will be correct or optimal.

• Combine ideas that it has seen separately but never in combination. For example, ChatGPT can write an answer to a coding question in the writing style of a specific author. 
• Exhibit a breadth of information. ChatGPT is trained on so much data that it has seen examples of most common situations and their potential variations. This enables it to give specific answers to niche questions or generalized answers based on more specific data.

What Are the Limitations of ChatGPT?

While ChatGPT outputs human-like sentences, and it’s easy to mistake its output as being powered by true intelligence, ChatGPT does have shortcomings. 

In describing the tool’s limitations, OpenAI explained that ChatGPT may occasionally “generate incorrect information” or “produce harmful instructions or biased content.” Industry publications have described ChatGPT as confidently wrong, exhibiting a tone of confidence in its answers, regardless of whether those answers are accurate. 

ChatGPT lacks the ability to fact-check itself or conduct logical reasoning. It often incorrectly answers questions and can be tricked relatively easily. Technologists have also noted its propensity to “hallucinate,” a term used to describe when an AI gives a confident response that is not justified by training data.

How ChatGPT Impacts Assessment Content

As a coding tool, ChatGPT excels at certain types of technical problems—but also has its limitations. A strong content strategy will be necessary to test your current coding challenges and prioritize the questions, and question types, that are less susceptible to AI coding support. 

ChatGPT has probably seen almost all known algorithms. But ChatGPT isn’t just able to answer these algorithm questions correctly. It’s also able to write new implementations of those algorithms, answer freeform questions, and explain its work.

As a result, ChatGPT can answer the following question types with reasonable accuracy:

• Well-known algorithms: It’s safe to assume that ChatGPT has seen and is able to answer all publicly available coding problems on platforms such as LeetCode and StackOverflow. If the algorithm appears in online forums or practice websites, ChatGPT will likely answer it correctly.

• Minor variations of problems. ChatGPT does well on variations that tend to add to the solution rather than change it in any substantial way. The system can, for example, easily reverse the order of an array of numbers.
• Multiple choice questions. When presented with a question and multiple potential answers, ChatGPT can usually identify the correct answer.

For hiring teams who administer coding challenges, that doesn’t mean you should necessarily avoid all questions that ChatGPT can solve. With the right protections in place, even questions solvable by AI can still be reliable. The key is to avoid questions that have answers so short that a plagiarism detection system can’t detect when a candidate has used a tool like ChatGPT. Even so, we are evolving our library with new types of content specifically designed with AI code assistance tools in mind.

Taking all of this into account, there are some actions you can take today to limit your hiring content’s exposure to the risk of plagiarism, including: 

• Avoid easily solved multiple choice questions
• Avoid simple prompts to solve for common or widely available algorithm variants
• Remove questions that require only a few lines of code to solve
• Use proctoring tools and plagiarism detection systems
• Combine coding tests with virtual interviewing tools to add empirical data to the hiring process

Ensuring Assessment and Hiring Integrity

In a world where humans and machines alike can write code, the ability to detect the use of AI-coding tools is invaluable. As such, employers increasingly turn to strategies and technologies that enable them to uphold the integrity of their technical assessments.

Assessment integrity has two core pillars: proctoring tools and plagiarism detection.

Proctoring Tools

One important component of ensuring assessment integrity is to build systems that provide the right proctoring capabilities. 

Proctoring is the process of capturing behavioral signals from a coding test, and its purpose is twofold. First, proctoring tools record data points that support plagiarism detection. Second, proctoring tools also act as a deterrent against plagiarism, as candidates who know that proctoring is in place are less likely to engage in such activity.

The key behavioral signals that proctoring tools often record include:

• Tab proctoring. Monitors if the candidate switches between tabs.
• Copy-paste tracking. Tracks if a candidate pastes copied code in the assessment.
• Image proctoring. Captures and records periodic snapshots of the candidate.
• Image Analysis. Analyzes webcam photos for suspicious activity.

Plagiarism Detection

In addition to proctoring tools, the integrity of an assessment also relies on plagiarism detection. In other words, the ability to flag when a candidate likely received outside help. 

The current industry standard for plagiarism detection relies heavily on MOSS code similarity. Not only can this approach often lead to higher false positives rates, but it also unreliably detects plagiarism originating from conversational agents like ChatGPT. That’s because ChatGPT can produce somewhat original code, which can circumvent similarity tests.

While the launch of ChatGPT caught many by surprise, the rise of LLMs has been a popular topic in technical communities for some time. Anticipating the need for new tools to ensure assessment integrity, HackerRank developed a state-of-the-art plagiarism detection system that combines proctoring signals and code analysis.

Using machine learning to characterize certain coding patterns, our algorithm checks for plagiarism based on a number of signals. Our model also uses self-learning to analyze past data points and continuously improve its confidence levels.

The result is a brand new ML-based detection system that is three times more accurate at detecting plagiarism than traditional code similarity approaches—and can detect the use of external tools such as ChatGPT.

Embracing Artificial Intelligence

As exciting as the launch of ChatGPT has been, LLMs with its capabilities are only the beginning. While it’s hard to predict the future, one thing is certain: AI technology is in a nascent state and will continue to grow at a rapid rate.

In the short term, the key to evolving your hiring strategy hinges on a renewed focus on content innovation and assessment integrity. By combining a strong question strategy with advanced proctoring and plagiarism detection, hiring teams can protect their assessment integrity and hire great candidates.

In the long term, we anticipate that artificial intelligence will redefine developer skills and, in the process, change technical hiring as we know it. 

At HackerRank, our mission is to accelerate the world’s innovation. As such, we welcome this new wave of technological transformation and will pursue innovative ideas that imagine a future of programming in an AI-driven world. 

Frequently Asked Questions

Can Your Plagiarism Detection System Detect Code From ChatGPT?

Yes. Our AI-enabled plagiarism detection system feeds several proctoring and user-generated signals into an advanced machine-learning algorithm to flag suspicious behavior during an assessment. By understanding code iterations made by the candidate, the model can detect if they had external help, including from ChatGPT.

When Will the Plagiarism Detection System Be Available?

The new plagiarism system is currently in limited availability, with plans for general availability in early 2023. If you would like to participate in our limited availability release, please let your HackerRank customer success manager know and we would be happy to enable you.

Can You Validate if My Coding Questions Are Easily Solved by ChatGPT and Provide Replacement Options?

If you would like assistance in verifying how ChatGPT responds to your custom coding questions, we can run a report and provide content recommendations based on the results. Please contact our HackerRank Support Team, who would be happy to help. 

Should I Avoid All Questions That ChatGPT Can Solve? 

No. HackerRank’s proctoring tools and plagiarism detection system can protect even solvable questions. Instead, avoid multiple choice questions and problems with very easy or short answers.

I Still Have Questions About ChatGPT. Who Should I Contact?

If you’re a customer looking for support on plagiarism and its impact on your business, you can contact your customer success manager or our team at support@hackerrank.com.

The post What Is ChatGPT? And What Does It Mean for Technical Hiring? appeared first on HackerRank Blog.