Leak to Us image
Investigations

Reboot practice for Hong Kong’s air traffic system runs counter to manufacturer’s recommendations

The Civil Aviation Department has been performing rotational restarts fortnightly on 47 air traffic workstations in a bid to prevent computer crashes

Workstations for Hong Kong’s troubled air traffic system have required more frequent reboots than recommended by its manufacturer, according to an official user guide obtained by FactWire.

FactWire revealed in May that the Civil Aviation Department (CAD) had started performing rotational restarts fortnightly on 47 air traffic workstations and weekly on six other workstations since February, in a bid to prevent computer crashes and sluggish performance of the system.

But the efforts were apparently in vain after it emerged earlier this month that glitches had intensified since April, prompting additional reboots during peak hours.

CAD had previously said that regular reboots were part of its ‘routine maintenance procedure’ recommended by the system’s manufacturer Raytheon.

However, a copy of AutoTrac III System Administrator’s Guide written by Raytheon has now come to light, detailing in about 300 pages steps and procedures on how to operate and service the air traffic management system.

According to Section 2.8 Housekeeping, restart of workstation software and workstation reboot are included in the 11 preventive and general maintenance procedures, along with other measures such as periodic archiving, purging of log files and directed switchover.

But restarts are only recommended to take place every 30 days as part of regular maintenance of the system and should only be performed during non-busy hours to minimise potential side effects.

Recommendations in the administrator’s guide run counter to the current practice adopted by CAD, which had admitted to rotationally rebooting 47 workstations every fortnight and six other workstations every week. 

Since late April, CAD has also been restarting some workstations and their software during peak hours from 7am to midnight despite the manufacturer’s guideline, although the department had said that additional reboots only occurred during period of ‘relatively low air traffic flow’.

In a statement on November 9, CAD said: ‘Major overseas ATCCs (air traffic control centres) also in general regularly reboot individual workstations based on their needs, which is part of their routine housekeeping procedures.’

The department also told FactWire earlier in June that airports in Singapore, Shanghai and Beijing would restart certain air traffic workstations every fortnight when required.

However, information shows that all three airports CAD raised as examples use air traffic management systems developed by French company Thales, instead of Raytheon, which made Hong Kong’s AT3.

The Civil Aviation Authority of Singapore did not respond to a request for comment, but a source knowledgeable of its operations said although some air traffic workstations at Changi Airport did require regular reboots after commissioning of a new air traffic management system due to performance issues, the practice was temporary and only lasted four to five months.

Restarting some workstations as necessary is also different to CAD’s current practice of regularly rebooting all workstations, he said.

According to sources, workstation rebooting was originally adopted by CAD as a temporary measure for use in non-busy hours.

After six incidents of sluggish performance since the system’s full commissioning last November, Raytheon at the time suggested rebooting some workstations as a preventive maintenance measure.

The first reboot took place on 26 November last year to prevent sluggish performance, which was followed by preventive reboots of more than ten other workstations on 2 December after another four cases of sluggishness. The practice continued on 26 December, 24 and 26 January when one, four and seven workstations were restarted respectively.

MISSING IMAGE
Timeline of workstation reboots in the past year (FactWire graphic)

Reboots then became part of a routine maintenance procedure after CAD drew up a schedule in February for rebooting all workstations rotationally every fortnight – a practice that has continued ever since.

Sources said the performance of the workstations did improve for a while, but glitches like sluggishness, frozen screen and unresponsiveness to commands resurfaced in April, causing additional reboots, with most happening in peak hours (see table below).

 In response to FactWire’s revelation on 9 November that glitches had caused about 20 reboots between April and October as well as of the details of 14 incidents, CAD denied any occurrence of ‘system crash’, but said ‘there were five confirmed cases of slight sluggishness of the screen display’.

It added that radar screens did occasionally experience sluggishness while mice and keyboards were not responsive to command, but insisted some cases FactWire revealed – without specifying which – did not happen according to the department’s record.

MISSING IMAGE
FactWire has learnt of the details of 19 manual hardware or software restarts of 9 workstations. Most of the incidents happened during peak hours from 7am to midnight. (FactWire graphic)

Frontline air traffic controllers said CAD had ‘conned the public with their wording trick’ by narrowly defining ‘system crash’ as ‘frozen radar screen or suspension of operation’ and disregarding situations such as unresponsive screen or sluggish performance.

A source also confirmed that all 14 incidents in the summary were reported to the department after maintenance staff inspected and rebooted the problematic workstations, but since the system failed to record some of the cases, CAD regarded those cases as ‘unsubstantiated’ and did not refer them to Raytheon for further investigations.

Routine maintenance normally include measures such as archiving and inspecting of hard disks which would not interfere with operations, according to the source, while it is also rare for other air traffic management systems to adopt monthly rebooting of workstations as a permanent maintenance procedure.

‘Bi-weekly restarting of every workstation is just a very desperate way of doing preventive maintenance,’ he said. ‘But even with such preventive measures, the problem still surfaced, the system’s reliability is questionable.’

MISSING IMAGE
Fact-checking CAD’s arguments (FactWire graphic)

這則報道獲以下新聞媒體採用:

蘋果日報   東方日報   頭條日報   明報   香港01   香港電台

Other stories

LeaveHomeSafe app has built-in facial detection module, government admits

李家超李文龍合照 Photo of John Lee and Gilbert Lee together

Business connections of John Lee’s sons with CE electors revealed

李家超 John Lee

John Lee’s first two properties incurred over $8m profit in the 90s

First instance of Wuhan Huoshenshan blueprint with restricted exterior access being built in Hong Kong’s Lok Ma Chau Loop

omicron-vaccine

CoronaVac limits Hong Kong’s protection against Omicron, even if entire population were to be vaccinated

omicron-vaccine-3rd-dose

Government puts little emphasis on local experts’ advice to opt for Comirnaty booster

relaunch hong kong

Surveyed Japanese react unfavourably to Hong Kong in $44m rebrand research

relaunch hong kong

Negative views on Hong Kong’s democracy and freedom revealed by 2020 publicity analysis

FW-PIX-2021-LEGCO-ELECTION-PRIVILEGED-1-864x500

41 Privileged voters’ power is 7,200 times a single-vote holder’s under new LegCo electoral system

Government’s broken promise: Original clock not required in the reconstructed old Star Ferry Clock Tower

Records show long hours of unplanned shutdown caused by ‘human factors’ in Taishan Unit 1’s first year of operation

17 High-salt/sugar foods qualify for government’s salt/sugar-reduction label scheme

Lands Department contradicts itself about SOGO’s possible land term violations

CCTV installed in two primary school’s classrooms but parents say they do not know

SOGO CLUB failed to follow its ‘members only’ license requirements for years