Alternatively please email our support address support@opencrm.Co.UK

Apologies for any inconvenience this may cause.

We have implemented some redundancy measures to correct the problem some customers have experienced with connectivity. All services should now be back online for all customers. If you are still experiencing problems, please do call us on 08452 303083.

—

Some customers may be experiencing difficulty in connecting to their OpenCRM services at present.

This is due to a connectivity issue in the area of the data centre. We are currently investigating this. Please check back soon.

It would appear that there is a Virus outbreak ongoing at the moment. Our engineers and online services team are working on this currently, all emails will be rigorously tested multiple times for viruses before releasing.

If you are using any of our Virus protection services this may cause a delay in emails being delivered.

On the 1st February we experienced high queues within our incoming mail servers. This resulted in delays in customers receiving emails to their mailbox.

Our engineers identified an issue which occurs only at times of high load on our mail storage platform. We worked through the night in conjunction with our storage suppliers to introduce more capacity.

Wednesday 2nd February

Mid-day: our email platform experienced exceptionally heavy load and action was required to maintain our email service and avoid any server downtime. During this period, access to POP3 was limited. While this was not a decision we made lightly, it meant that we were still able to maintain normal email service to a large number of our customers.

We have also now freed up additional storage capacity through other means, which will take effect from tonight.

13:30: We have restored POP3 access to all mailboxes. However, we have placed a connection limit on the number of open concurrent connections in order to maintain stability. As a result during peak times some customers connecting to their mailboxes using the POP3 protocol may find access intermittent.

14:00: While the connection limit for POP3 connections is still in place, a very large number of our customers should experience no impact at all. A small number of customers may experience brief losses of connectivity.

Thursday 3 Feb

10:50: The mitigating work completed yesterday does not appear to have been successful. As volume of traffic across our mail platform has increased the number of concurrent users has again caused some issues. Our engineers will be conducting maintenance to our mail platform to ensure that stability can be maintained.

We would like to apologise for the inconvenience caused by this issue and hope to resume normal service as soon as possible.

12:30: Maintenance work has been completed and the platform was brought back online. After consultations with our storage suppliers we have taken the decision to conduct further maintenance to our mail platform. This has also been completed satisfactory. All access to emails will remain to be intermittent while we clear our email queues and traffic returns to normal levels. Once the email queues have returned to normal levels we will be able to re-assess the impact of this maintenance work and take action accordingly.

13:20: Traffic levels remain to be high across our mail platform as we work to clear the backlog of mails caused during our maintenance work. While traffic remains high we are still seeing connections to mailboxes being dropped as they hit our maximum connections limit. This will result in a continued intermittent connectivity when accessing emails, which should reduce throughout the day.

16:45: Due to the complexity of our email architecture, some customers will find access to their email more intermittent than others. Further maintenance is planned for tonight, when load across our mail platform is light. In the meantime, while there is currently no change in status to this issue all technical resources available are continuing to be used in resolving this as soon as possible.

21:00 – Customers using Standard and Advanced mailboxes should find that mail is being sent and received normally. Connection to mailboxes remains intermittent, but this should reduce during the evening. Webmail performance is reduced, with occasional disconnections and timeouts.

Due to essential maintenance, the mail platform will be unavailable for a period of approximately 20 minutes at from 22:00 GMT. Mail sent during this time will be queued and sent once the maintenance is complete.

Friday 4th Feb

Following the maintenance work last night, the email platform is more stable today, but we are still experiencing some intermittent connectivity issues.

Customers using Standard and Advanced mailboxes should find that mail is being sent and received normally. However, connection to mailboxes remains intermittent and Webmail performance is likely to be reduced.

Our engineers are investigating this issue as a priority, and we apologise for the inconvenience caused.
12:00: Our mail platform has returned to normal service levels. Our engineers will continue to monitor the performance of the mail platform over the course of the next few days.

No email has been lost as a result of this issue.

Monday 7th Feb

14:25 – Email access has been restored for the majority of our customers. We are currently migrating all remaining customers affected by this issue to a new storage platform and mailboxes will become accessible as soon as they have been migrated. This action is likely to result in a much improved service for the majority of affected customers.

We have been seeing intermittent issues with traffic crossing the London Internet Exchange (LINX) peering network over the last two days, which has been causing low levels of packet loss when routing traffic to some UK locations.

LINX are currently investigating what appears to be a faulty port within one of their 160GB Fibre trunks. Whilst this is on-going, we have downed our LINX peering and all traffic will now be routing over alternative paths.

We will be keeping a close eye on the situation and will only re-enable the LINX peering once we are certain that all issues have been resolved.

Please be aware that the issues we’re referring to lie with the LINX peering network, and not something we have direct control over. This is why we have taken the action to disable the connection until further notice.

Should you have any questions or concerns, please raise a ticket or give our support team a call.

Database connectivity on a single server in the data centre : 9:12

One of the Business Level servers seems to have a problem with database connectivity and this is being looked at as a priority and we will post more details here shortly. This will effect a small handful of Business level clients, please accept our apology for the downtime. This server has been under performing since 8:43 this morning.

Full connectivity has now been restored but our engineers are monitoring the situation for any further problems

————–

We are aware of some connectivity issues which may be preventing some customers from being able to access their OpenCRM systems at present.

Our engineers are looking at this and expect to have this resolved very shortly.

We are currently experiencing some issues receiving calls which may result in some people having difficulty getting through to us. We are working on this and should have this resolved soon. Please email support@add-ons.co.uk with any queries in the meantime.

Update 22/09/10 22:39 – Our DNS providers have taken some steps to reduce the problems caused by the DDOS and things are now looking more stable with access to most addresses on our domains now working. We will continue to monitor this until we are informed a full solution has been put in place. We will be implementing some measures to ensure that we are more resilient to this sort of attack in the future.

Update - Our 3rd party DNS provider is currently under a severe denial of service attack which is affecting access to OpenCRM services at the moment. Although all servers are up, the DNS provider is intermittantly unable to resolve the OpenCRM URL’s which is making accessability to the systems intermittant.

We are working to resolve this as soon as possible. We are also investigating alternative providers to prevent this problem from happening again in the future.

———————

We are aware of some DNS resolution issues that may affect some customers access to their OpenCRM system at the moment.

This is not a problem at the OpenCRM data centre which is up and running OK but our engineers are investigating what we can do to circumvent this problem for the limited number of customers affected.