Diamond found himself on Tin Pan Alley when he was barely out of his teens, leaving his scholarship to New York University to take a 16-week writing gig. While he was undeniably talented, Diamond struggled to write for other people, which meant he rarely held down a job as a staff songwriter for long. He was in and out of employment for seven years, existing for one of them on one sandwich a day. Eventually, he set up shop himself, putting a piano in a room on Broadway and finding new inspiration.

Diamond collaborated with writing duo Ellie Greenwich and Jeff Barry, and came up with three hits during their first writing session together – Solitary Man, Cherry Cherry and I Got the Feeling (Oh No No). Solitary Man became Diamond’s first chart record, and its modest success, “was enough to turn me from an unknown songwriter pounding the streets for eight years to a guy who has a record on the charts,” he explained to the LA Times.

The song is now a firm favourite among Diamond fans, and its meaning has become more transparent to the singer over time. “I wasn’t trying to write anything about myself necessarily at the time, I thought it was just a nice idea to write a song about a solitary guy,” he explained. “It wasn’t until years later, when I went into Freudian analysis, that I understood that it was always me.”

I’m A Believer, 1966

The jaunty love song was a hit for The Monkees, but actually written by Diamond – a rarity considering he floundered in his early songwriting career after struggling to write hits for other musicians. Although Diamond recorded the song and still occasionally performs it live, he had never intended to keep it just to himself. Before The Monkees snapped it up, Diamond had tried to sell it to country singer Eddie Arnold.

Diamond was was “thrilled” that his song brought The Monkees such success – it is one of a handful of songs to have sold 10 million physical copies worldwide – and has said that “they kind of saved my career”. But his record company were less so. Diamond said his label boss “went through the roof because he felt that I had given Number One records away to another group,” in a 2008 interview with Mojo magazine. “I couldn’t have cared less because I had to pay the rent and the Monkees were selling records, and I wasn’t being paid for my records.”

Shilo, 1967

Diamond’s father was a dry goods salesman, and moved the family around to make the best living possible. As a result, Diamond had been to nine schools before he was 16, leaving him without the opportunity to make meaningful friendships. Instead, he made an imaginary friend, Shilo, whom he paid tribute to on this deeply personal song in 1967. Diamond later said that he wish he hadn’t painted such a harsh portrait of his father on the song.

Shilo has become a fan favourite, although it was never one of Diamond’s most commercially successful hits. This became a bone of contention between the singer and songwriter/producer Bert Berns, his employer at Bang Records. Diamond was keen to stray away from the chirpy pop he’d been creating for Berns, and wanted to put out Shilo as a single. Berns refused, and Diamond walked out on his contract. The singer later claimed that Berns, who died aged 38, threatened him, and believes that he organised the deployment of a stink bomb in Greenwich Village venue The Bitter End to punish Diamond.

Brooklyn Roads, 1968

Diamond’s first single for MCA Records was also one that would define his songwriting, with a lyrics that told a visual story of his upbringing. Although he lived in Wyoming as a child, he spent most of his time in different parts of Brooklyn, and it was here that he started to take guitar lessons and began songwriting after an inspiring session with Pete Seeger at summer camp in upstate New York.

While some of Diamond’s songs were written purely to channel emotion, Diamond says this one was “consciously autobiographical. I wanted to try and capture what it was like growing up in Brooklyn – for myself, just to have that feeling for a moment.” It took on a profound resonance when Diamond performed at New York’s Madison Square Garden in 2008, when he played against a backdrop of images from his childhood.

Sweet Caroline, 1969

The infectious chorus of Sweet Caroline is reliant on one particular thing: the three syllables of the name “Caroline”. Diamond plumped for the name after realising that he couldn’t quite stretch out that of his wife, Marcia, to fit the notes. It helped, of course, that in the mid-Sixties Caroline was the youngest daughter of president John F Kennedy and his wife Jacqueline. Diamond said that, a couple of years before the song came out, he had been moved to write about the girl after seeing an “innocent, wonderful picture” of her in a magazine. “I immediately felt there was a song in there,” he told Associated Press.

Diamond has also said that Sweet Caroline was the result of divine intervention. He posted on Reddit a couple of years ago: “There’s something about that song – from the moment i wrote it, I felt a connection with a higher force. It was a totally un-premeditated – the song just came to me when I needed it most, when I was at my lowest point ever in my career, when I thought my career was over, that song came to me and gave me back my career and my life. I think that song is an act of God, and I couldn’t be more thrilled that I had a little part in making that happen.”

Jonathan Livingston Seagull, 1973

Not an individual song, but a collection of them: Diamond’s Grammy-winning soundtrack outsold the film it accompanied sixfold ($12 million to $2 million, according to one report) and eased the concerned minds of the executives at Columbia who had just made a new deal with Diamond.

But what’s even more unlikely than Jonathan Livingston Seagull’s success is its creation. While he was working on the record, a Hare Krishna knocked on Diamond’s door. The musician invited him in and showed him his work. The pair met every day for six weeks, with Diamond renting him an apartment and a car. When his new friend invited Diamond to travel to India with him to live in a cave, the star reluctantly turned him down – he had Seagull to finish – but paid for his airfare regardless.

Beautiful Noise, 1976

Diamond’s 10th album was also billed as a comeback of sorts – his first in a couple of years (a normal break for many artists, but Diamond’s previous nine records had been released in eight years – and saw him veer into new musical territory. But the whole thing was inspired by a trip to New York, which Diamond took with his daughter Marjorie (one of two from his first, short-lived marriage in the late Sixties). They stayed at the Sherry-Netherland Hotel which looks over 5th Avenue, and watched a Puerto Rican Day Parade go by. It proved to be inspiration for a hastily constructed song. As Diamond remembered: “My daughter Marjorie looked out the window and said, ‘What a beautiful noise,’ and it just hit me like a bolt. I ran over and wrote it down. My parents were there, and I just sang the song into a tape recorder.”

Love on the Rocks, 1980

Another example of what happens when Diamond gets involved with a film soundtrack. While The Jazz Singer, which Diamond starred in alongside Laurence Olivier, bombed at the box office, soundtrack number Love on the Rocks spent four weeks at number two on the US charts after being released as a single.

The song was originally called Scotch on the Rocks and, bizarrely, had a reggae beat (which is ironic, when you consider what UB40 did to Red, Red Wine, without ever thinking it was a Diamond song). Diamond collaborated with the film’s other songwriter, Gilbert Bécaud, to work on the verses that saved it.

Some two decades later, comedy rock band The Darkness took their own spin with album track Love on the Rocks with No Ice. And, bizarrely, Diamond even knows what form it would take as a cocktail: “Probably U-Bet chocolate syrup, I love it, mix some milk in there, put some seltzer in there, mix it up, and you got the greatest, most refreshing drink you’ll ever have. An egg cream cocktail is what I would end up with. It’s a New York concoction and everybody in the city knows what it is.”

America, 1981

Another from The Jazz Singer’s soundtrack. The patriotic song closed the film, but has also been used for grander purposes, such as by Michael Dukakis in his 1988 presidential campaign and during the 1996 Olympics, which were hosted in Atlanta.

Diamond wrote it for his grandparents, who were immigrants to the country. “It’s my gift to them, and it’s very real for me,” he told the LA Times. “It wasn’t thought out or intellectualised, just sheer emotion. In a way, it speaks to the immigrant in all of us.” In the wake of the September 11 attacks, Diamond changed the lyrics of the refrain when performing the song, from “They’re comin’ to America” to “Stand up for America.”

I am… I said, 1988

Diamond’s most soul-baring tribute to loneliness was inspired by a particularly brutal audition – for Bob Fosse’s film about the foul-mouthed comedian Lenny Bruce. As Diamond told Rolling Stone, studying for the part and rehearsing the script was a transformative experience for him: “He was just saying all those things I had been holding in. It was all the anger that was pent up in me. Suddenly here I was, speaking words that I had never spoken before. These violent monologues of his, and the way he acted. And I went into therapy almost immediately after that.”

During a break from the screen test, Diamond took a few minutes to wallow, convinced that he had done miserably. This made for the beginnings of Diamond’s first Grammy nominated song: “I was in my dressing room with my guitar. It just came out.” It wasn’t, however, as easy as that. I am… I said took a further four months to finish. In 1992, Diamond called it “by far the most difficult song I have ever written – and probably the best song I have ever written.” He maintains that it is “easily” his favourite song to perform – even though he doesn’t do it often.