The university’s home directory server went offline affecting numerous Galaxy jobs.
|11:02||We received a mail from the HPC team that the central Uni Freiburg fileserver was experiencing issues resulting in an outage.|
|16:30||Some jobs were still hung and could not recover due to this outage. The UseGalaxy.eu Admin team began discussing possible options.|
|17:18||We took the decision to reboot affected hosts as processes could not be killed. This affected a number of jobs unfortunately that were not rescheduled automatically.|
Killing the affected hosts was not an optimal solution however it was the most efficient. Jobs should have been rescheduled and re-run automatically but it seems they were often not, so we will be investigating and correcting issues with our Galaxy job configuration.