| Listing of main services location |
| RS3(US) | FR |
| j1 | j2 | j3 | j4 | j5 | j8 | fr.timduru.org |
fursuit database
user websites
user galleries
mysql DBs
postgresql DBs
forums
ftp server
mrtg
webalizer web sites stats
user ssh
|
fursuit archive
rss feeds (fpod, archive..)
|
mail server
nameserver
mailing lists
mysql DBs
|
File hosting
|
stream server
ftp user
user ssh
|
Wikifur multilanguages
|
fursuit archive (Round Robin with j2)
stream server (Round Robin with j5)
animal archive
baltosource
roilionsource
Other websites
ftp
mysql DBs
mrtg
webalizer web sites stats
backups
|
Server Status, Maintenance / Downtime
(dates are based on GMT+2 time)
STATUS DATE TIME DURATION SERVICES DESCRIPTION
DONE 20091106 02:00 ~06:00? j8:* Squid lock down, needed restart
DONE 20091015 11:05 00:03 rs3 reboot after kernel update
DONE 20090907 22:40 00:35 fr:mysql lock due to file system full
DONE 20090817 09:55 00:03 fr:main reboot
DONE 20090729 08:50 00:03 fr:main reboot after main system upgrade
DONE 20090614 17:35 15:30 fr:main reboot after main system upgrade
Server not coming back online.
Ticket open
DC doesn't seem to be working before 10am on Monday.. great.
server booted back
DONE 20090614 16:10 00:15 fr:chroot distrib upgrade, a few seconds / minutes of downtime for services as they are upgraded
DONE 20090613 23:03 ~00:10 j8,j1 upgrades+APC problem
DONE 20090610 18:48 00:03 rs3 Kernel update (security patch) + reboot
DONE 20090606 19:05 00:02 rs3 Kernel options tweak + reboot
DONE 20090606 18:35 00:03 rs3 Upgrade to PAE kernel to see the 4GB #Try2
=> OK
DONE 20090606 17:30 00:23 rs3 Upgrade to PAE kernel to see the 4GB
Problem with new kernel not seeing some filesystems
=>back to old kernel
Need to see later on for new upgrade.
DONE 20090606 15:50 00:10 rs3 RAM upgrade
DONE 20090603 23:05 00:25 fr Reboot after upgrade.
Server not coming back, ticket open.
fsck required.
DONE 20090512 23:45 00:40 rs3 Datacenter network routing issue, rendering certain IP ranges unreachable
DONE 20090508 15:10 05:50 rs3:snmpd:cpu MIB fix missing cpu MIB stats
DONE 20090508 15:30 00:15 j3:mail server upgrade spam modules + perl
DONE 20090508 15:00 00:10 rs3 kernel upgrade + restart + OS upgrade + restart
DONE 20090508 14:20 00:01 j8:squid restart after upgrade
DONE 20090508 14:10 00:01 j8:mysql restart after upgrade
DONE 20090508 13:45 00:05 j8 OS upgrade + restart
DONE 20090508 13:10 00:02 j3 Restart after software rebuild
DONE 20090508 12:45 00:05 j3 OS upgrade + restart
DONE 20090507 10:15 00:05 fr:misc Software Upgrades:apache, mysql, libs etc..
DONE 20090505 19:05 00:10 j1 OS Upgrade + restart
DONE 20090505 12:00 07:10 j1:mysql:innodb tables mysql server version bug
The problem was affecting only innodb tables,
that is mostly the galleries and the fursuit database
Non innodb db tables were not affected
DONE 20090504 21:40 00:01 j2 Restart after software rebuild
DONE 20090504 21:02 00:02 j2 OS upgrade + restart
DONE 20090504 20:30 00:02 j4 Restart after software rebuild
DONE 20090504 19:55 00:02 j4 OS upgrade + restart
DONE 20090504 18:05 00:01 j5:DSS Restart DSS after Upgrade to 6.0.3
DONE 20090504 18:00 00:01 j5 Restart after Software rebuild
DONE 20090504 13:30 02:40 j5 OS upgrade
DONE 20090323 11:40 00:02 j* System update requiring reboot
DONE 200902XX XX:XX 00:10 j* A few services might get stopped for a few minutes as datas are transferred to the new HD
DONE 20090204 07:20 06:05 j* New HD firmware upgrade, due to Seagate bug
4.5 hours later, they ask me how to burn the damn iso to boot from to perform the update!
Dear technician, it Would be better to unplug the server only once you are ready maybe...
5 hours later, they still didn't figure out how to boot from the iso..
I even tested it myself and burnt it and it boots fine, just click, burn , boot!!
6 hours later they have finally flashed the firmware, and server is back online they say.
well nope !
Dear technician, if the system doesn't ping, and tells you at boot that there is a
problem and asks for fixing the problem by entering single user mode,
it means the server is not back online!
Anyway I connected through the KVM and fixed the problem and now we're back , grrr.
DONE 20090128 13:27 00:10 j1 home directories + misc move between HDs
DONE 20090124 10:20 00:01 j2 Data transfer to new HD + service stop
DONE 20090123 01:35 03:00 j* Hardrive add-on
DONE 20081222 14:15 00:02 fr reboot after bootloader reconfigure
DONE 20081222 09:50 03:55 fr server didn't restart - ticket open
[13:30] Datacenter reply: bootloader seems to be corrupted
=>asked them to reinstall it.
[13:45] Server back online.
DONE 20081222 09:45 00:03 fr system upgrade
DONE 20081205 14:59 00:10 fr system upgrade
DONE 20081022 16:45 01:20 j1:meerkatforum migration to phpBB3
DONE 20081021 23:45 00:01 j4:* various system upgrades, services might go down for a few minutes every now and then
DONE 20081021 23:10 00:02 j5:* various system upgrades, services might go down for a few minutes every now and then
DONE 20081021 22:40 00:01 j2:* various system upgrades, services might go down for a few minutes every now and then
DONE 20081021 20:40 00:02 j*:* kernel upgrade + reboot
DONE 20081021 19:45 00:02 j8:* various system upgrades, services might go down for a few minutes every now and then
DONE 20081021 13:30 00:05 j*:* various system upgrades, services might go down for a few minutes every now and then
DONE 20080915 20:50 00:30 fr filesystem full.
mysql table repairs needed.
Repair done.
DONE 20080715 11:25 00:05x3 j* shpgperproc increase + kernel tuning due to new jail and processes + reboots
DONE 20080611 16:35 03:30 FR* FR provider optical fiber cut. again!
DONE 20080526 21:10 00:15 j1:postgresql postgresql cleanup
DONE 20080526 14:40 00:02 j4 sync data + move to the new server
DONE 20080525 16:31 00:05 j3 sync data + DB + move to the new server
DONE 20080514 11:55 04:20 FR* FR provider optical fiber cut.
DONE 20080426 14:42 00:50 j1 sync data + DB + move to the new server
DONE 20080424 19:10 00:02 j2 sync + move to the new server
DONE 20080330 18:20 00:35 j* HD replacement
DONE 20080330 16:15 02:05 j* One of the new HD is dying.
Data recovery from the dying HD.
HD replacement scheduled
DONE 20080330 14:45 01:30 j* Offline FSCK+Drive testing
DONE 20080330 14:28 00:10 j* Server lock. Rebooting
DONE 20080330 14:00 00:15 j* Server lock. Rebooting
DONE 20080330 13:25 00:20 j* Server lock again..
DONE 20080330 12:40 00:25 j* Disk crash, followed by a server crash.Rebooted
DONE 20080313 21:05 00:03 j* reboot after kernel upgrade
DONE 20080312 11:41 00:03 j* reboot after upgrade
DONE 20080311 11:20 00:05 j1:http,mysql DB migration to new partition
DONE 20080311 10:50 00:07 j1:* Home directories partition migration to new HD.
DONE 20080304 20:10 02:33 j* Memory upgrade + Hard drive4 add-on
DONE 20080304 05:05 01:30 j* Hard drive2 replacement+server swap and reconfigure
DONE 20080112 08:20 00:28 j1:FranceFurs forum phpBB3 migration
DONE 20080109 10:55 00:30 fr:mysqlbaltosource table repairs
DONE 20080109 09:25 00:30 fr:ALL server upgrade + reboot +fsck
DONE 20071003 10:45 00:12 fr:mysql table repair
DONE 20071002 22:00 12:00 fr:ALL local flood on the datacenter network rendered the server almost unreachable
Server got rebooted first because they thought it was a problem on the server.
=> mysql tables need manual repair
DONE 20070920 19:32 2:30 rs:ALL server not responding, ticket open.
took them 2.5 hours to reboot the server, grr.
DONE 20070822 13:29 0:03 fr:ALL reboot after kernel upgrade.
DONE 20070817 17:25 0:12 fr:ALL +1GB RAM addon.
DONE 20070716 12:00 1:30 j*:ALL NIC replacement.
DONE 20070715 13:30 3:00 fr:ALL Data center network down.
DONE 20070709 04:30 3:30 fr:general slowdown The return of the mirroring guy, this time putting the server into swap mode,
rendering it very very slow.
Tweaked the mirroring lock values that were still too high, as he was still hammerring the server
it seemed to work in keeping the server out of swap mode.
Then banned his IP, even though it's a dynamic IP...
DONE 20070706 15:00 3:00 fr:apache heavy mirroring took apache down=>max num of connections per IP decreased
will see if it happens again.
DONE 20070706 13:58 0:20 fr:ALL 160 GB HD replacement by 500GB HD
DONE 20070701 15:30 0:25 baltosource migration to 2nd server
DONE 20070628 ~07:00 ~0:30 j*:ALL Router upgrade
DONE 20070312 11:40 2:00 j*:ALL Server back online in single user mode
All services down, as tech support requires
that I leave the server in single user mode the time they investigate
No investagtion started after more than 2 hours >:|
Can't wait for them forever ! I rebooted in multiuser mode without waiting for them.
Server rebooted ok in multi user mode,
I'll reopen a ticket if the problem comes back, when there is someone available dammit!
DONE 20070312 10:51 0:50 j*:ALL HD3 problem again => reboot
Server, not coming back online. Ticket opened with tech support.
DONE 20070219 16:10 0:25 j1:pgsql forums pgsql php lib problem after upgrade => reinstall
DONE 20070205 15:15 0:05 j*:ALL forced reboot after some HD problem => HD to monitor.
DONE 20061231 15:30 0:10 j1:Mysql Forums repair tables with invalid status
DONE 20061231 14:40 0:35 j1:ALL upgrade j1 system OS (4.11=>5.5)
DONE 20061231 01:15 1:15 j3:ALL upgrade j3 system OS (4.11=>5.5)
DONE 20061123 13:55 3:30 j*:DB Disk full problem
DONE 20061007 13:25 0:05 j*:ALL System upgrade
DONE 20060910 0:30 fr:ALL New 160GB HD upgrade
DONE 20060908 19:55 0:05 j*:ALL old 200GB HD removal
DONE 20060908 19:00 0:05 j*:ALL old 200GB HD transfert
DONE 20060907 09:10 0:30 j*:ALL New 400GB HD install
DONE 20060630 0:02 j2:ALL upgrade j2 system OS (4.11=>5.5)
(server might experience slight slow downs, as I compile the system before switching)
DONE 20060629 0:02 j4:ALL upgrade j4 system OS (4.11=>5.5)
(server might experience slight slow downs, as I compile the system before switching)
DONE 20060628 0:02 j5:ALL upgrade j5 system OS (4.11=>5.5)
(server might experience slight slow downs, as I compile the system before switching)
DONE 20060627 j*:stats webalizer websites stats reinstall
DONE 20060626 09:35 j*:mrtg installed back
DONE 20060625 18:40 0:13 j*:ALL fsck of all the disks,
now that everything should be ok and that the tech won't be touching the server anymore,
I prefer to force a check disk to be sure we start on good base, I think it's better after all the wild reboots from the techs.
DONE 20060625 12:30 05:50 j*:ALL Scheduled HD removal (old system HD)
see the history of problems below to see the stupidity of the guys removing the HD
and for your pleasure here is the best-of
DONE 20060625 15:35 j*:ALL they are removing the HD ..
DONE 20060625 14:30 j*:ALL server back, everything is fine from my point of view, they will try *AGAIN* to remove the old system HD
Hopefully it will work this time and they won't remove the wrong HD.
DONE 20060625 13:52 j*:ALL server now running a check disk, they want me to check the conf before they try *AGAIN* to remove the stupid old HD
DONE 20060625 13:50 j*:ALL I had told them to put back the old drive to check if it was booting back, they said they tried it and it was not working
now I get another message from the person in charge ( not the tech) that he checked himself, and after putting back the old drive
they were supposed to remove it's botting again !!! wtf ! What the heck are they doing
DONE 20060625 13:20 j*:ALL it seems they broke the server again after removing the HD, the server doesn't boot anymore, great
the server was rebooting perfectly before they touched it ! dammit,
they are investigating... great.
DONE 20060625 01:40 xxxx j1,j5:ssh connection fixed
DONE 20060624 21:45 01:05 j*:ALL stupidly locked myself out of the server by inserting the firewall rules,
server is beeing unlocked by tech support..
sorry about that, I think I really need sleep..
DONE 20060624 20:26 0:01 j5:ALL Data sync after reinstall
DONE 20060624 19:52 0:03 j4:ALL Data sync after reinstall
DONE 20060624 19:11 0:00 j2:ALL Data sync after reinstall
DONE 20060624 16:45 0:05 j3:ALL 1 Data sync +config after reinstall
DONE 20060624 15:40 0:10 j1:ALL 1 Data sync +config after reinstall
DONE 20060624 j*:ALL
11:22 All systems should be back to normal now.
I still need to reconfigure and upgrade some of the components, it will go down for a few minutes at a time when I do so.
10:41 main user conf is back, still many things to reconfigure but for the moment websites should be mostly back.
I will have to take them down for a few minutes from time to time to reconfigure / upgrade things.
07:50 system restore finished, I'm reinstalling all
04:39 New ticket for restore open.Great, I have nothing better to do than to wait for them at almost 5am ..
04:35 they still say I have to reopen another ticket for a restore , grrrr 3 hours lost for nothing
03:30 They closed the ticket saying that it was incorrect wtf??
01:35 I've ordered a priority 2 hours restore of the current server,
but it will most likely not help on the reboot problems,
but the server will be back temporarily after I've reconfigured / installed everything back.
00:45 No new server in stock, might take up to 3 days to get a new one
00:10 Bad news, DC told me that the server is not booting anymore.
DONE 20060623 XXXX j*:ALL again and not coming back , DC will check the hardware in single user mode.. might take some time.
DONE 20060623 0:10 j*:ALL reboot again.
DONE 20060622 2:20 j*:ALL reboot still, didn't come back online, manual fsk
DONE 20060622 0:20 j*:ALL scheduled memory + power cables change
DONE 20060622 0:15 j*:ALL again..
DONE 20060622 0:10 j*:ALL again..
DONE 20060621 0:20 j*:ALL server down again, tech investigating...
DONE 20060621 1:10 j*:ALL server rebooted and had to run a fsck
DONE 20060620 0:10 j*:ALL server rebooted again
DONE 20060620 0:20 j*:ALL server rebooted +fsck
DONE 20060325 0:10 j1:php php recompile/upgrade
DONE 20060325 0:02 j1+j3:mysql mysql upgrade
DONE 20060313 0:25 j*:ALL FreeBSD upgrade
DONE 20051008 0:45 j1:phpBB
post&search phpBB problems in the post & search functions with php5.0.5 => required to edit source files of all installations.
DONE 20051008 0:25 j1:php5-modules Scheduled php-modules upgrade
DONE 20051008 0:01 j1:php5 Scheduled php upgrade
DONE 20050905 1:15 j*:ALL Scheduled HD removal, problems after the removal required them to FSCK all left disks (340GB)