 |
|
| Home | About MC | Academics | Admission | Alumni | Information | Sports | Student Life | Search | |
|
|
|
| Home | About MC | Academics | Admission | Alumni | Information | Sports | Student Life | Search | |
|
Listing of Information Systems Unplanned Outages |
||
|
This page lists unplanned Information System service outages. The most recent is on top. For more detailed information please contact the Information Systems Office. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
| Issue: | Internet Outage |
| Dates: | Wed, November 15th, 2006 - 1:00 PM through 1:45 PM |
| Details: | Our Internet service provider informed us that the outage was caused by a router failure that affected multiple states. |
| Issue: | Administrative database - very slow performance |
| Dates: | Tues, August 21st, 2006 - 7:30 AM through Wed, August 23rd, 3:00 PM |
| Details: |
An administrative software upgrade was
implemented on Monday evening and the system was tested by the
office most affected by the software changes. The tests showed
no anomalies. Early Tuesday morning we started to receive reports of a performance problem as we accessed the database. We consulted the software manufacturer to see if the upgrade could have caused the problem. The problem was vigorously researched by the software manufacturer and by Monmouth College Information Systems staff. Wednesday, around 12:30 PM, it was discovered that a core table index was lost. Following that clue we discovered more missing table indexes. We replaced the identified indexes and the system performance was restored. The investigation continues to determine exactly how the indexes were removed. |
| Issue: | Email slow, delayed delivery |
| Dates: | Fri, August 4th, 2006 through Fri, August 11th |
| Details: |
About Friday, Aug 4, 2006 our email
system started receiving more junk (SPAM) email than normal.
Eventually, the number of email grew so large that the incoming
queue started to fall behind. When the problem was fully researched
and understood, there were close to ½ million emails in the incoming
queue. The end result was long delays for valid email messages. The fix was to temporarily move the backlog of messages and allow the server to quickly deliver any new incoming email. That occurred on Tuesday, Aug 8, 2006. The next step was to manually filter the ½ million messages by valid email addresses and finally to place those messages back into the queue. Friday, August 11, 2006 we added the delayed mail back into the delivery system. The result was successful but late email delivery. This problem is not unique to Monmouth College. For anyone interested in the gritty details, here is a link to a whitepaper (from HP Labs) that describes the problem and suggests some solutions. http://www.hpl.hp.com/techreports/2004/HPL-2004-5R1.pdf |
| Issue: | Email Down, some applications missing, some students could not logon. |
| Dates: | Mon, April 3rd, 2006 - 7:30 AM through 1:30 PM |
| Details: |
A critical domain server failed with a
corrupt database. The email system is dependent upon the
domain server for authentication processes. Consultation with the
software manufacturer was required to repair the server to avoid
data loss. This process took about six hours Hardware upgrades during the weekend forced data to be moved from older to new servers. The data transfer took longer than expected. This resulted in some applications not being available and some students could not log onto the domain. All services were restored by 3:00 PM and there was no data loss. |
| Issue: | Power failure - loss of all services |
| Dates: | Thurs, Feb 16th, 2006 - 4:00 AM through 11:00 AM |
| Details: |
Because of an ice storm there were power fluctuations in our area.
Sometime around 4:00 AM this morning (Thursday) our computer /
network power support system suffered a catastrophic failure. This
resulted in a shutdown of all electronics that support our servers
and the main network. The problem was discovered around 6:00 AM and troubleshooting began. It was determined that the power system could not be quickly repaired. As a temporary solution, the power support system was bypassed and the servers and network were restarted. Services were reactivated and all services were restored by about 11:00 AM. However, the system is at risk until the power system is restored. Parts are on order and the system will be restored when they arrive. Replacing the parts will require a full shutdown of the system. We will notify the Monmouth College community as soon as we can determine the replacement time and date. |
| Issue: | Loss of Internet Connection - Office / Labs |
| Dates: | Thurs, Sep 15th, 2005 - 9:45 PM through Fri, Sept 16th, 6:30 AM |
| Details: |
A hub concentrator failed.
This equipment connects the office / lab network onto the Internet.
The hub was replaced. This did not affect the residence hall network. |
| Issue: | E-mail Down - DNS error |
| Dates: | Wednesday, April 13th, 2005 - 6:00 AM through 7:20 AM |
| Details: | Intermittent problems on main
network caused by DNS service failure. The most notable symptom was
the main e-mail system would not function properly. The DNS service
was restored. This did not affect the residence hall network. |
| Issue: | Residence Hall Internet Problems |
| Dates: | Wednesday, March 23rd, 2005 - 6:45 PM through 7:50 PM |
| Details: | Poor performance on the residence hall network possibly caused by a SYN attack. A SYN attack is usually caused by a virus infected computer sending malformed packets to a server. It is also possible that this could be a malicious act. The problem reoccurred intermittently during the remainder of the evening and early morning. |
| Issue: | Internet Outage |
| Dates: | Tuesday, March 15th, 2005 - 11:55 AM through 12:50 PM |
| Details: | Our Internet Service provider had a power failure in Macomb. Power was restored and the system was restarted. |
| Issue: | Application Server |
| Dates: | Monday, November 29th, 2004 - 4:50 PM through 5:35 PM |
| Details: | The main application server experienced a software failure. The exact reason for the failure is currently unknown. The system was restarted and is working normally. |
| Issue: | Student residence hall Internet access |
| Dates: | Friday, November 19th, 2004 - 12:01 PM through 12:10 PM (9 minutes) |
| Details: | A software reconfiguration forced a system reboot. |
| Issue: | Loss of access to administration system |
| Dates: | Tuesday, November 9th, 2004 - Wednesday, November 10th |
| Details: | A programming error resulted in a loss of connectivity to part of the administration system. The error was identified and repaired at 8:00 AM on Wednesday. |
| Issue: | External mail services unavailable |
| Dates: | Friday, November 5th, 2004 - Monday November 8th |
| Details: | Precipitated by a software failure in a Linux sendmail system. All incoming and outgoing e-mail was delayed. There was no loss of messages. Most messages were eventually delivered. Some e-mail bounced back to the sender. |
| Issue: | Wireless Connection in Hewes Library |
| Dates: | Saturday, November 6th, 2004 - Sunday November 7th |
| Details: | A switch hardware failure resulted in unreliable wireless connections in the basement of the Hewes Library. |
| Issue: | Communications line failure |
| Dates: | Wednesday, November 3rd, 2004 - Thursday, November 4th |
| Details: | One T1 telephone line failed about 4:00 PM
on Wednesday and was repaired about 2:00 PM on Thursday. This
resulted in a loss of performance on the residence hall network.
Please note that the T1 lines are managed by one or more telephone
companies. Monmouth College reports failures and must wait for
the telephone companies to respond. The charts below describe the outage and should be read from right to left. The 0 mark (red lines) represents midnight.
|
| Home | About MC | Academics | Admission | Alumni | Information | Sports | Student Life | Search | |
|
|
Monmouth College 700 E. Broadway - Monmouth, Illinois 61462 Phone: 309-457-2131 - Fax: 309-457-2141 Email: Contact MC Copyright © 2002-2004 Monmouth College ® - All Rights Reserved |
