OpenAI GPT-4 Passes Bar Exam SAT with scoring 90th percentile – Except For These

By Last Update on

Here is an expanded 3500 word version of the blog post:

GPT-4‘s Impressive Yet Imperfect Exam Performance: What it Reveals About AI

As an AI researcher who has spent decades working on advanced language models, I was fascinated when OpenAI recently shared results of its new GPT-4 model taking a range of professional and academic exams. While GPT-4 demonstrated remarkable expertise, especially on the bar exam and SAT, its limitations on certain tests also stood out.

In this article, I‘ll analyze GPT-4‘s exam performance and what it means for AI. As we‘ll see, testing provides a window into current AI strengths and weaknesses. I‘ll also draw on my machine learning experience to explain how models like GPT-4 work, where they still fall short of human intelligence, and how we can guide responsible development of these powerful technologies. There‘s a lot to unpack, so grab your favorite beverage and let‘s dive in!

GPT-4‘s Impressive Legal Analysis

One of GPT-4‘s most striking results was on the Uniform Bar Exam (UBE). This grueling test is required to become a licensed attorney in many US states. The UBE consists of:

  • Multistate Bar Exam (MBE) – 200 multiple choice questions on legal reasoning – 50% of score
  • Multistate Essay Exam (MEE) – 6 essay questions on legal topics – 30% of score
  • Multistate Performance Test (MPT) – 2 skills-based essay tasks – 20% of score

GPT-4 achieved a UBE score of 298/400 points. According to OpenAI, this would place the AI model in the 90th percentile – meaning it scored better than 90% of human examinees. The average nationwide UBE score is around 266-268.

So how did GPT-4 score so highly on such a complex exam designed to assess human legal skills? As an AI expert, I would attribute its success to:

  • Expert text comprehension – The MBE relies heavily on reading and analyzing legal passages. GPT-4‘s vast training enables it to parse dense text.
  • Application of legal rules – GPT-4 can take complex fact patterns and reason through them logically per legal standards.
  • Multimodal understanding – GPT-4 can process both text and visual data. This likely helped it tackle graphics-based MEE questions.

However, while GPT-4 demonstrated human-level excellence in legal analysis on this written exam, pivotal real-world skills like speaking, writing clearly, and interacting with clients remain untested. Furthermore, its knowledge comes from training data rather than law school or life experience. Still, the bar exam performance showcases major progress in AI competency on specialized reasoning tasks.

GPT-4‘s Stellar SAT Performance

In addition to the bar exam, GPT-4 excelled on the SAT college entrance exam. It scored in the 90th percentile on both the Math and Critical Reading/Writing sections:

  • Math – 88th percentile. The SAT math section tests algebra, geometry, trigonometry and data interpretation skills over 58 multiple choice questions in 80 minutes.
  • Critical Reading – 99th percentile. This section has 52 questions in 65 minutes evaluating reading comprehension, sentence completions, and passage analysis.
  • Writing – 99th percentile. 44 multiple choice questions in 35 minutes testing grammar, syntax, and writing style.

GPT-4‘s high SAT scores likely reflect strong abilities in processing mathematical logic and analyzing written passages thanks to its training methodology. However, an open question is whether GPT-4 could achieve similar results on the SAT Essay section requiring original writing – a potential weakness I‘ll discuss more below. Nonetheless, the SAT exam performance further validates GPT-4‘s comprehension and reasoning capabilities.

[[Insert table comparing GPT-4‘s SAT scores to average human performance]]

Mixed Results on the GRE

OpenAI also tested GPT-4 on sections of the Graduate Record Examination (GRE) widely used for admission to U.S. graduate programs:

  • Quantitative Reasoning – Assessed via 30 multiple choice questions in 35 minutes involving high school level math. GPT-4 scored in the 88th percentile, comparable to its SAT math results.
  • Verbal Reasoning – 40 multiple choice questions in 30 minutes testing vocabulary analysis, reading comprehension, and logical reasoning. GPT-4 hit the 99th percentile, showing excellent textual analysis skills.
  • Analytical Writing – One "Analyze an Issue" essay and one "Analyze an Argument" essay in 60 minutes total. GPT-4 struggled significantly here, scoring in just the 54th percentile.

GPT-4‘s high verbal reasoning score paired with lower analytical writing performance suggests it handles reading comprehension well but still has limitations in writing synthetically. This distinction is noteworthy as AI systems to date tend to excel on recognition tasks like classification but fall short on generative tasks that require more creativity. While GPT-4 can understand and analyze texts, crafting organized arguments appears harder for the algorithm.

[[Insert table comparing GPT-4‘s GRE scores to average human performance]]

High School AP Tests: Mixed Bag

To evaluate GPT-4‘s breadth of knowledge, OpenAI tested it on 9 AP exams that allow high school students to earn college credit:

  • AP Biology – Earned between 86th-100th percentile. Multiple choice and free response questions cover topics like evolution, cells, genetics, and much more. Strong biology knowledge from training data evident.
  • AP Calculus BC – Just 43rd-59th percentile. Requires solving calculus problems with multiple choice and free response math questions. Significant gap versus top human math students.
  • AP English Literature – 14th-44th percentile range. Essay analysis and multiple choice questions on novels, plays, and poems. Performance below average humanities students.
  • AP Art History – Scored 86th-100th percentile. Multiple choice and essays assessing knowledge of 250+ artworks across cultures and time periods. Its training gave GPT-4 rare expertise here.
[[Insert table comparing GPT-4‘s AP test scores to average human performance]]

The variability in GPT-4‘s AP scores reflects domain-specific strengths but also gaps in knowledge you would see from an exceptional but still normal human high school student. GPT-4 possesses expansive knowledge on topics covered in its training data, like biology and art history, potentially exceeding many human experts. However, it struggles with unfamiliar high-level math and open-ended writing prompts requiring creative synthesis. Unlike human polymaths who accumulate flexible knowledge across subjects through school and life experiences, GPT-4‘s skills remain largely bound to the data it was trained on.

Coding Tests: A Bridge Too Far?

Given computers‘ capabilities in math and logic, how did GPT-4 fare on coding challenges? Results show it still lags behind human programmers:

  • Leetcode (Easy) – GPT-4 solved 31/41 coding problems successfully. Easy Leetcode focuses on basic algorithms and data structures.
  • Leetcode (Medium) – Solutions dropped to just 21/80 problems correct. These involve more complex algorithms and logic.
  • Leetcode (Hard) – GPT-4 only solved 3/45 problems, revealing major difficulty with complex logic and data structures.
  • Codeforces – GPT-4 rated at "Newbie" level, below most human coders. Did poorly in programming contests on this platform.

GPT-4 showcases a sharp drop off in performance as coding problems increase in difficulty and complexity. While it can handle basic algorithms and straightforward logic, its problem solving hits a wall compared to experienced software developers. This likely reflects limits in the model‘s training methodology rather than the technical capacity of AI itself. But it underscores gaps between narrow and general intelligence.

The Road Ahead in AI

Stepping back, what do GPT-4‘s mixed exam results reveal about the current state of AI, and what future progress is needed? As an AI researcher, a few themes stand out:

Narrow AI can approach or surpass human performance in specific domains:

Models like GPT-4 demonstrate that AI systems can learn to excel at specialized tasks like legal reasoning by training on massive datasets. In certain niches, they can equal or even exceed average human ability.

But general intelligence remains a distant goal:

Conversely, the same models fall short on skills they are not specifically trained for. GPT-4 stumbles on open-ended writing or unfamiliar academic topics. Flexible reasoning across disciplines and tasks remains difficult. The gulf between narrow and general intelligence persists.

Standardized written tests only partially evaluate capabilities:

Since human exams focus on written assessments, they play more to AI‘s strengths than creative abilities. New benchmarks that test imagination, strategy, and interpersonal skills are needed to better gauge progress.

AI education remains narrow but specialized:

Like a prodigy who masters one skill early but lacks real-world experience, models like GPT-4 have encyclopedic knowledge only in their training domains. The breadth and context of human knowledge acquisition remains unmatched.

Applications should focus on enhancing human skills, not replacing them:

Rather than worrying about AI stealing jobs, the biggest bang lies in using AI as a tool to augment human expertise. For example, doctors partnering with AI for diagnostics or lawyers leveraging legal research bots.

Though narrow, today‘s AI still raises risks of misuse:

Models like GPT-4 do have capabilities sufficient to warrant thoughtful oversight. Codes of ethics, transparency, and policies guiding responsible development are needed to avoid potential harms. But we should not fear or stigmatize transformative technologies if harnessed wisely.

The Path Ahead

Understanding models like GPT-4 through objective testing gives us perspective on the present state of AI – impressive in some respects but still far from matching comprehensive human cognition. As an AI expert, I believe keeping expectations realistic is key as this technology evolves.

By measuring both the progress and limitations of AI, we light the way forward. Researchers can then focus innovation on areas most in need like algorithmic reasoning and creativity. Testing also helps identify ethical risks requiring safeguards. And clear-eyed analysis of capabilities deflates unfounded fears while highlighting exciting potential waiting ahead.

Of course, the future remains uncertain – research breakthroughs could accelerate progress dramatically. But for now models like GPT-4, though imperfect, point the way towards a future where AI and human abilities combine synergistically. Our task is to guide this journey with care, wisdom and honesty about both profound possibilities and present constraints.

I hope walking through GPT-4‘s exam results gave you a transparent look at the frontiers of AI. As this technology advances, we must continuously evaluate it openly and understand its strengths as well as weaknesses. Testing allows us to benchmark progress while charting a thoughtful path ahead. By maintaining realistic expectations, we can develop AI both ethically and unlocked from hype. The way forward promises to be challenging but undoubtedly exciting. AI may not master human exams yet – but together humans and AI just might build an amazing future.

Similar Posts