A couple of hours south of the marinas of Istanbul in the middle of the Sea of Marmara sits Imrali island, a no-go area sealed off by the Turkish state. The island is Turkey’s most high-security prison – its the equivalent of Alcatraz or Robben Island in South Africa – adapted to incarcerate one man, Abdullah Öcalan, the founder of the Kurdistan Workers’ party (PKK) – an armed group of Kurdish fighters engaged in an insurrection against the Turkish state for 30 years.

Public enemy No 1 to the Turks, lionised by the Kurds, Öcalan has been demonised by Ankara for most of the 14 years he has been in solitary confinement on the island. The prime minister, Recep Tayyip Erdogan, even said recently he would have liked to have seen Öcalan executed.

In recent weeks that has changed, raising hopes of a breakthrough in the quest to settle one of the world’s longest-running and most debilitating ethnic conflicts, which has cost up to 40,000 lives over 30 years.

Turkey’s intelligence chief, Hakan Fidan, has been visiting the island to cultivate Öcalan.

“Fidan and Öcalan have managed to understand each other,” said Ayla Akat, a Kurdish MP who is one of the few politicians to have visited the prisoner.

The inmate’s brother Mehmed has become a visitor. Erdogan announced the provision of a TV for the guerrilla leader. The government is keen to reveal how many books Öcalan has read and the fact he plays football and basketball on the island where he has been joined by five prisoners.

In short, Turkish demonisation of Öcalan has given way to a process of humanisation paving the way for peace talks that some, including Turks and Kurds, liken to the UK-Irish negotiations that led to the Good Friday accords.

“The novelty is not that the state is talking to Öcalan – that’s happened before – but that they are admitting it,” said Cengiz Çandar, an analyst of the conflict. “There are new unprecedented elements that have raised expectations of a breakthrough. That can also be very dangerous.”

Of the estimated 30 million Kurds in Iraq, Iran, Syria and Turkey, around half are in Turkey, concentrated in the south-east but also dispersed to the big cities in the west of the country, not least Istanbul, home to about 3 million.

The momentum towards the “Imrali process” – as the incipient peace talks are being dubbed – has been provided by a series of tragedies and catastrophes both regional and within Turkey, apparently bringing both sides to conclude they have fought themselves to a stalemate.

“There has to be a political solution. The armed struggle has run its course. But the PKK will continue to fight if there is no political solution. Both sides know that’s the case,” said Akat.

The past 18 months have been one of the most vicious periods of the 30-year insurgency, leaving 900 dead, the heaviest casualty rate since Öcalan was captured in Kenya in 1999, the International Crisis Group said.

In the same period, the Turkish authorities jailed thousands of Kurdish activists, sparking a hunger strike last autumn involving up to 600 inmates. Just when it became critical in October, Öcalan ordered the strike to end and everyone complied.

It was a persuasive demonstration of the leader’s power after 14 years in jail, making it plain that if Erdogan wanted to sue for peace, he would need, indirectly, to talk to Öcalan.

“The war can go on without Öcalan but there can be no peace without Öcalan. Everyone understands that,” said Mazlum Dinç, one of the Kurdish leader’s lawyers.

“Öcalan is the one person who can bless a compromise agreement. He remains the paramount figure,” said Hugh Pope, the crisis group’s analyst in Istanbul.

If the bloodshed, jailings, and stalemate are pushing the parties to the negotiating table, the other big factor is regional, where the dynamic favours the Kurds over the Turks, supplying a further reason for Erdogan to soften.

As a result of the Iraq war, the Kurds of northern Iraq in effect enjoy home rule. An estimated 3,000 PKK fighters are holed up in the Qandil mountains of Iraq, with a similar number inside Turkey. But the most recent game-changer has been the civil war in Syria on Turkey’s south-eastern border. The PKK’s Syrian cousins now control tracts of north-east Syria and also might expect to win regional autonomy in a postwar settlement.

The Turkish and Syrian Kurds have gained control of about 435 miles of the border.

“It’s Bashar Assad’s revenge against Erdogan,” said a senior European diplomat. “He has ceded north-east Syria to the Kurds to cause trouble for Ankara.”

Others say the Kurdish control of the region is simply a result of the war. “Turkey’s Middle East policy has crashed, exposing it to the Kurds. It needs a deal with the PKK to be stronger in the region against Baghdad and Tehran,” said Pope.

The rush of boat trips to the prison shifted up a gear at the weekend when three members of the Kurds’ Peace and Democracy party (BDP) – to the PKK what Sinn Féin was to the IRA – were allowed to visit Öcalan for a tightly monitored eight hours to obtain his thoughts on peace talks and returned with a long letter from the leader.

The talk in Ankara and Istanbul is of the PKK calling a ceasefire next month during the Kurdish new year celebrations, of a possible release of Turkish hostages held by the PKK, and of the fighters retreating into the Iraqi mountains while laying down their arms from August.

According to leaks in the Turkish press on Thursday, Öcalan told his visitors the peace process had to succeed, since the alternative was “war and chaos”, warning that a force of 50,000 Kurdish insurgents would escalate their fight against the Turkish state. It was not clear who leaked the transcript and why but the incident only thickened the air of conspiracy and manipulation surrounding the process.

If Öcalan still rules the roost with the Kurds, the same is true of the other side where decision-taking stops and starts with Erdogan, unassailable in Turkish politics as he approaches 10 years in office. He aims to emulate Russia’s Vladimir Putin next year by swapping the premiership for an executive presidency under a new constitution.

This week Erdogan has been loudly anticipating the prospect of PKK disarmament and making disparaging remarks about the Kurds, hardly the behaviour of someone seeking to build trust across the communal divide. The latest example of Erdogan raising hackles came on Friday when the US secretary of state, John Kerry, visiting Turkey, took the prime minister to task for remarks calling Zionism a crime against humanity.

Erdogan is famously inscrutable. He is refusing to say what is in a peace process for the Kurds or what may have been promised to Öcalan by his messenger Fidan.

This lack of candour is feeding suspicion and recrimination on the Kurdish side, and complaints from the main Turkish opposition party, which broadly supports the peace moves.

“There’s a new generation of Kurds that has known nothing but war,” said Hayri Ates, a Kurdish politician. “Their villages were destroyed, all the unsolved murders and disappearances. They’re destitute. And they blame all their grievances on Turks. It’s an angry generation. Now the country is polarised and Erdogan’s party is hegemonic. But it is going to have to talk to the Kurds on equal terms.”

The scepticism about Erdogan’s good faith is reinforced by statements from the government side.

“You have to take into account the sensitivities of non-Kurdish citizens,” the deputy prime minister, Hüseyin Çelik, said. “We have to manage public opinion. Öcalan is a political prisoner who still has influence over his organisation. But Öcalan and the PKK can’t get anywhere by killing people. You cannot shake hands with a fist.”

Western diplomats in Ankara doubt the seriousness of the parties, saying while both sides have fought to a stalemate, they are not weary enough to abandon hopes of prevailing by force.

There are strong suspicions that Erdogan is being driven less by a strategic vision but by tactical scheming aimed at concentrating his political power through the new constitution.

Nonetheless, the public choreography of the peace process is different from anything that has gone before, also encouraging a wary optimism even on the Kurdish side.

“There have been many attempts at peace talks since 1993,” said Mesut Yegen, a Kurdish sociologist in Istanbul and a historian of the conflict. “This time is different. For the first time Erdogan is a partner you can trust. And Turkey has to act while Öcalan is still able to deliver.”

Nihat Ali Özcan, a Turkish counter-terrorism expert in Ankara, thinks time and the political dynamics in the region are on the Kurds’ side. “Öcalan has a lot of time on the island, while Erdogan has a very expensive watch,” he said.

But he says decades of brutality on both sides have engendered an unforgiving climate which will be hard to change. “We can tolerate 500 deaths a year. It’s considered normal.”

While it is unclear what Erdogan is offering, if anything, the Kurdish demands amount to a straightforward package of civil rights denied to them since the modern republic was founded 90 years ago: education in their own language; recognition in the proposed new constitution that Turkey is not a republic of “Turks” but also of Kurds and the other 40 ethnic minorities in the country; election laws that lower the threshold for entering parliament currently designed to minimise Kurdish representation; greater decentralisation and regional government. The Kurds will also demand that Öcalan be allowed to swap his island isolation for a form of mainland house arrest.

“Turkey has had an indefensible policy towards the Kurds since 1925 and it has blown so many chances,” said Pope. “This is simply about equal rights and justice.”

He added the potential for a breakthrough had seldom been better. “I’ve never seen the situation so pregnant with possibility, he said.”

Erdogan and Öcalan appear to be deadly enemies locked in a fateful embrace, with Turkey’s future hinging on whether they are bold enough to take a risk for peace.

The price of failure will be high, the likelihood of a return to even worse bloodshed in a conflict calculated to have cost Turkey up to $450bn (£300bn), according to government figures.

The prize of a settlement could seal Erdogan’s place in history as the greatest national figure since Kemal Atatürk, the republic’s founder, and see him awarded the Nobel peace prize.

“It’s not the last chance and it’s not the best chance,” said Çandar. “But it’s a good chance.”