Automated Events
Ongoing deployment operation can contain may different operations including snapshots, ser- vice backups, data source backups, scaling events (increasing or decreasing), and auto-recovery.
The two most critical events for a deployment are auto-scaling and auto-recovery. The diagram below shows some of the processes behind this activity.
Auto-Recovery
Auto-recovery is an event that is triggered when there is an unexpected server failure in the cloud. enStratus monitors servers as a matter of course and when a failure is detected, a recovery event is triggered.
What happens during a recovery event depends somewhat on the role of the server in the deployment.
In the case of an application server, the following event timeline is followed:
- Failure detected.
- New server started in the same zone as the failed server using parameters defined in the server group.
- Failed server removed from any load-balancing by stopProxy.
- Failed server removed from any database dependencies by revokeDatabaseAccess. Once the server is started, the volume that was orphaned by the failed server will be re-attached to the new server.
- If the orphaned volume cannot be reattached, a new volume will be generated by taking a snapshot of one of the volumes attached to a currently running server in the server group and creating a volume from that snapshot. The resulting volume will be attached.
- If there are no running servers in the server group, a volume will be created from the most recent snapshot on record and the volume will be attached to the new server.
- If there are no snapshots, a new volume will be created and attached. The service will be installed as if starting the deployment from scratch.
- Once the volume is attached, enStratus calls /enstratus/bin/startService for each service contained on the volume.
- Recovered server added to any load-balancing by startProxy.
- Recovered server added to any database dependencies by grantDatabaseAccess.
In the case of a database server, the following event timeline is followed:
- Failure detected.
- New server started in the same zone as the failed server using parameters defined in the server group.
- Once the server is started, the volume that was orphaned by the failed server will be re-attached to the new server.
- If the orphaned volume cannot be reattached, a new volume will be generated by taking a snapshot of one of the volumes attached to a currently running server in the server group and creating a volume from that snapshot. The resulting volume will be attached.
- If there are no running servers in the server group, a volume will be created from the most recent snapshot on record and the volume will be attached to the new server. If there are no snapshots, a new volume will be created and attached. The service will be installed as if starting the deployment from scratch.
- Once the volume is attached, enStratus calls /enstratus/bin/startService for each service contained on the volume.
- If any application servers depend on the database server, enstratus-configure will be called on each application server with an updated enstratus.cfg file containing information about the new database server.
Next, let's look at Deployment Scaling.
Updated: 08-01-2011: