Remember the scene where Roy Scheider's character (Dustin's brother, in the movie) meets the villain (played by Laurence Olivier) for the first time, and pays with his life? You may recall the unusual, stair-like red fountain/sculpture in the background. It's supposed to be somewhere in New York.  It isn't.

That red sculpture in the background is named Double Ascension, by artist Herbert Bayer, and you'll find it at the Arco Plaza (now known as City National Plaza) at 555 S. Flower Street, in downtown L.A.
That's in the plaza between the two towers, off the west side of Flower Street (between 5th & 6th streets).

Here is an edited video of that scene from "Marathon Man", highlighting the sculpture:

The sculpture has been seen in a number of other films & TV shows, including 1990's "Pretty Woman", 2007's "Rush Hour 3", 2005's "Fun with Dick & Jane", and in several episodes of the TV show "Heroes", where it was once again passed off as a New York location (this time, called "Kirby Plaza"). It's even briefly glimpsed during an aerial shot in a 2008 episode of "Smallville" ("Prey"), where it's supposed to be Metropolis.

And the Los Angeles filming didn't stop with the fountain.

Remember the library scenes where Dustin's character first meets Elsa? That was supposed to be New York's Columbia University. And while they did use some exterior shots from that campus, all of the interior library shots were filmed inside Doheny Library, on the USC campus, right here in L.A.

(Dustin also filmed scenes at this same library for his breakthrough role in 1967's "The Graduate".)

But wait, there's more.

Early in the movie, we glimpse the Uruguay jungle, and the hideout of our Nazi villain (Olivier), as he changes his appearance and readies to venture of hiding, for a trip to America.

Did they shoot that in Uruguay? Sorry.  Southern California again.

This time, it was the Los Angeles County Arboretum, at 301 N. Baldwin Ave., in Arcadia (right across the street from the Santa Anita Racetrack).

The white watchtower we glimpse (where the guard spots a small boat arriving and rings the bell) is none other than the "Fantasy Island" house, where little Tattoo rang the same bell at the start of every episode, while delivering his trademark line: "Da plane, boss! Da plane!"

In real life, it's the Queen Anne home of Lucky Baldwin, located next to a lake/lagoon at the Arboretum (which used to be the Baldwin Estate). The lake, with its tropical foliage, has often been used for jungle scenes in movies such as "Tarzan" and "The African Queen". (And it's open to the public, if you'd like to visit.)

Finally, the water works, the location of the big finale, where Dustin finally gets the drop on Olivier...