= Amazon EC2 = [[TOC]] Amazon's Cloud provides a flexible platform to deploy Eden scalably. The costs aren't fixed & can be difficult to predict, despite their [http://aws.amazon.com/calculator calculator], but are competitive, especially in Singapore, which is a good base for the Asia Pacific region. Users who are using the free tier: Remember - after 750 hours, your trial will end and the credit card on file will be charged based on the rates shown in EC2. You can prevent these charges by closing the AWS account from the [https://portal.aws.amazon.com/gp/aws/manageYourAccount/ AWS account management page]. == Getting Started == If you haven't already, create an Amazon AWS account through [http://aws.amazon.com/free/?sc_channel=PS&sc_campaign=AWS_Free_Tier_2013&sc_category=aws_cloud_computing&sc_publisher=Google&sc_medium=Brand_Core_AWS_E&sc_content=31496064882&sc_detail=Amazon%20-%20aws&sc_matchtype=e&trk=AWS_Free_Tier_2013/ their site].[[BR]] After creating your account, head to the [https://console.aws.amazon.com/console/home?#/ management console] and set up an instance of EC2. Instructions to do this can be found in the [http://aws.amazon.com/documentation/ec2/ Amazon EC2 Documentation]. == Regions & Zones == Amazon supports multiple Regions in order to provide a service closest to your users. * Namespaces of Instances, Volumnes & Snapshots are unique only within a Region. * Within each Region, there are a couple of Availability Zones to allow spreading the risk across different facilities. * Volumes are located within a specific Availability Zone * Bandwidth transfers are free within an Availability Zone == Instance Size == * The free starter 'micro' instance is flexible as it can run both 32-bit & 64-bit Operating Systems. * The normal production 'small' instance can only run 32-bit. * Larger production instances can only run 64-bit, so can't have the exact same image used. == Instance Persistence == * EBS-backed instances have persistent storage even whilst powered down, which is very useful. * For DB I/O performance increase can stripe multiple EBS * monitoring data available to see if this is the issue == AMI == * Recommend using the AWS !MarketPlace Debian image (as this has a sufficiently large HDD to start with) * In time we may provide pre-built "Sahana Eden" AMIs (some old unmaintained ones may be available in some regions) == Elastic IP == Each time you start an instance up, it will be assigned a new IP ('Public DNS') although this can be overcome using an Elastic IP * Remember to set up Reverse DNS for your Elastic IP to be able to send emails reliably: * https://aws-portal.amazon.com/gp/aws/html-forms-controller/contactus/ec2-email-limit-rdns-request * If you have a free EC2 instance, be sure to release your Elastic IP if you shut down your instance. IPv4 addresses are a "scarce resource" so Amazon will charge you for wasting one if you keep it assigned to your instance while you are not using it. == Authentication == === SSH === This can provide an early stumbling block. * Each instance created needs to start with a unique SSH keypair * When setting up an instance, be sure to safely download the private key. * In order to get the public key (needed by SecureCRT for instance) then you need to login using CLI & retrieve it (username 'admin' for the AWS !MarketPlace Debian, username 'root' for some other Images): {{{ ssh -l admin -i private.pem cat ~/.ssh/authorized_keys }}} SecureCRT needs the private key storing as & the public as (all on one line) SSH will also need to be enabled in your Firewall policy (restrict the IP source, if possible) === Recovering From a Lost Keypair === If you lose your keypair then you need to: * Create a new keypair in the AWS console & download the generated private key * Stop the instance * Create an AMI from this instance * Wait for the AMI to be ready * Launch a new instance using this AMI * Re-associate the Public IP * Delete the old instance * Deregister the AMI * Delete the snapshot used to create the AMI Thanks to: http://itkbcentral.blogspot.co.uk/2011/07/replace-lost-key-pair-existing-aws-ec2.html == Add Swapfile == You can add swap from a swap file in order to improve performance (especially on a Micro instance): {{{ sudo su - dd if=/dev/zero of=/swapfile1 bs=1024 count=524288 mkswap /swapfile1 chown root:root /swapfile1 chmod 0600 /swapfile1 swapon /swapfile1 # Make persistent across reboots cat << EOF >> "/etc/fstab" /swapfile1 swap swap defaults 0 0 EOF }}} == Add Storage == If you need an additional disk for Storage then configure a volume in the AWS console, attach as /dev/sdb1, then in Linux: {{{ sudo su - fdisk /dev/xvdb1 n (accept defaults) w mkfs.ext4 /dev/xvdb1 tune2fs -m 0 /dev/xvdb1 # Remove 5% reservation for reserved blocks mkdir /data cat << EOF >> "/etc/fstab" /dev/xvdb1 /data ext4 defaults,barrier=0 1 1 EOF mount /data }}} == Install == * Copy the installation and configuration scripts into the launched instance (assuming [wiki:InstallationGuidelines/Linux/Server/CherokeePostgreSQL Cherokee & PostgreSQL]): {{{ wget http://eden.sahanafoundation.org/raw-attachment/wiki/InstallationGuidelines/Linux/Server/CherokeePostgreSQL/install-eden-cherokee-postgis.sh chmod a+x install-eden-cherokee-postgis.sh wget http://eden.sahanafoundation.org/raw-attachment/wiki/InstallationGuidelines/Linux/Server/CherokeePostgreSQL/configure-eden-cherokee-postgis.sh chmod a+x configure-eden-cherokee-postgis.sh }}} * Run the install-eden-cherokee-postgis.sh script. [Note: This step takes about 10min - grab a coffee] {{{ sudo su - ./install-eden-cherokee-postgis.sh }}} If you wish to update your site from an alternate github repo this can be done using: * ConfigurationGuidelines#SwitchtoanalternateGitHubrepo == Configure == Run configure-eden-cherokee-postgis.sh to configure the instance: {{{ sudo su - ./configure-eden-cherokee-postgis.sh }}} * Add your FQDN to /etc/hosts to ensure emails are accepted by all remote mailers: {{{ vim /etc/hosts 127.0.0.1 host.domain host localhost /etc/init.d/exim4 restart }}} [UserGuidelines/Admin/Configuration Admin Guide] - especially read how to set the sender & approver emails == Add a Test site == This script requires at least 4Gb on the main disk {{{ sudo su - wget http://eden.sahanafoundation.org/raw-attachment/wiki/InstallationGuidelines/Linux/Server/CherokeePostgreSQL/add_test_site.sh chmod a+x add_test_site.sh ./add_test_site.sh }}} NB This script has an issue & the file /etc/cherokee/cherokee.conf needs to be manually edited to fix the lines wrapping for Source 1 (fix welcomed!) == Add a Demo site == This script requires at least 6Gb on the main disk. This script assumes that a Test site has already been installed {{{ sudo su - wget http://eden.sahanafoundation.org/raw-attachment/wiki/InstallationGuidelines/Linux/Server/CherokeePostgreSQL/add_demo_site.sh chmod a+x add_demo_site.sh ./add_demo_site.sh }}} NB This script has an issue & the file /etc/cherokee/cherokee.conf needs to be manually edited to fix the lines wrapping for Source 1 (fix welcomed!) == Grow the Diskspace == The initial disk space on some images is just 1GB. If you have this, then this should be grown to 4Gb (don't just size the volume to 4Gb to start with as the image only uses 1Gb of it!) * this is still within the 10Gb free tier. * 4Gb is needed for Prod & Test instances. If you just need a test then 3Gb is sufficient. You can do this using the AWS EC2 Console or else you can do it via the CLI === CLI Tools === To use any of the AWS CLI tools on your own machine to remotely manage instances, then you need to generate a unique X.509 Certificate per account. This can be done from the 'Security Credentials' page within your account. ==== CLI Management ==== There are extensive CLI tools available to manipulate your instances. * Java CLI for !Windows/Linux * http://aws.amazon.com/developertools/351 * http://serktools.com/2009/05/19/setting-up-ec2-command-line-tools-on-windows/ * http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/ * Python: http://libcloud.apache.org === CLI Script === Edit the settings as-indicated as you proceed through the script {{{ # Settings for Instance set EC2_URL=https://ec2.us-east-1.amazonaws.com set ZONE=us-east-1c set DEV=i-950895f1 set OLD=vol-31f5a35d # Stop Host ec2stop %DEV% # Create a snapshot ec2-create-snapshot %OLD% # Record the snapshot ID set SNAPSHOT=snap-63f89d08 # Create new volume from snapshot ec2-create-volume -z %ZONE% --size 4 --snapshot %SNAPSHOT% # Record the new Volume ID set NEW=vol-a9c2a3c4 # Attach new volume as secondary ec2-attach-volume -i %DEV% %NEW% -d /dev/sdb1 # Delete Snapshot (if no data in yet) ec2-delete-snapshot %SNAPSHOT% # Start Host ec2start %DEV% # Re-attach the Public IP # Login mkdir /mnt/data echo '/dev/xvdb1 /mnt/data ext3 defaults,noatime 0 0' >> /etc/fstab mount /mnt/data resize2fs /dev/xvdb1 umount /mnt/data shutdown -h now # Unattach volumes ec2-detach-volume -i %DEV% %OLD% ec2-detach-volume -i %DEV% %NEW% # Attach volume as boot ec2-attach-volume -i %DEV% %NEW% -d /dev/sda1 # Attach old volume for /var/log ec2-attach-volume -i %DEV% %OLD% -d /dev/sdb1 # OR Delete old volume #ec2-delete-volume %OLD% # Start Host ec2start %DEV% # Re-attach the Public IP # Login df -h # Use the old partition for /var/log (to avoid DoS) vi /etc/fstab /dev/xvdb1 /var/log ext3 noatime 0 0 mv /var/log /var/log_old mkdir /var/log mount /var/log mv /var/log_old/* /var/log rm -rf /var/log/bin/ rm -rf /var/log/boot/ rm -rf /var/log/dev/ rm -rf /var/log/etc/ rm -rf /var/log/home/ rm -rf /var/log/initrd.img rm -rf /var/log/lib/ rm -rf /var/log/mnt/ rm -rf /var/log/media/ rm -rf /var/log/opt/ rm -rf /var/log/proc/ rm -rf /var/log/root/ rm -rf /var/log/sbin/ rm -rf /var/log/selinux/ rm -rf /var/log/srv/ rm -rf /var/log/tmp/ rm -rf /var/log/usr/ rm -rf /var/log/var/ rm -rf /var/log/vmlinuz rm -rf /var/log_old }}} == Building AMIs for easier deployment == If your region doesn't yet have a Sahana AMI in, then it is easy to create one which allows easier deployment of future instances for both you & others. The only cost to you is a small amount of time to publish it & then using up your 1Gb Snapshot allowance within the free tier. * Install the instance as-above * Don't run the configure script * Shutdown the EBS instance. {{{ shutdown -h now }}} * Go to your aws console for the current region at [https://console.aws.amazon.com/ec2/home | EC2 Console] * Select the debian instance you just prepared in the "Instances" * Select " Create Image (EBS AMI) " in the "Instance Actions" drop down. [[Image(http://eden.sahanafoundation.org/raw-attachment/wiki/InstallationGuidelines/Amazon/create-image.png, 75%)]] * Choose an Image name - Lets say - "Sahana Eden" and fill in a description. [[Image(http://eden.sahanafoundation.org/raw-attachment/wiki/InstallationGuidelines/Amazon/create-image-config.png, 75%)]] * Click "Create this Image" * Go to the "AMIS" menu and select "Owned by me" and "All Platforms" in the Viewing dropdown. [[Image(http://eden.sahanafoundation.org/raw-attachment/wiki/InstallationGuidelines/Amazon/ami-built.png, 75%)]] * Wait for about 5 minutes and hit refresh for the AMI you just built to appear. * Select the AMI and click the Permissions button - Select public, to make the image public. [[Image(http://eden.sahanafoundation.org/raw-attachment/wiki/InstallationGuidelines/Amazon/ami-permissions.2.png, 75%)]] Keep Templates as EBS Volumes as this is cheaper than Snapshots == Troubleshooting == To troubleshoot any errors in installation of EC2 visit its [http://aws.amazon.com/documentation/ec2/ documentation]. If you encounter problems installing eden on the EC2 instance, you can contact us via [http://webchat.freenode.net/?channels=sahana-eden&uio=d4/ IRC] or the [https://groups.google.com/forum/#!forum/sahana-eden/ mailing list]. == Installing Deployment Coapp on EC2 == * Once you've registered for Amazon's Web Services, login to the [https://console.aws.amazon.com/ec2/v2/home EC2 Management Console]. You'll be greeted with the EC2 dashboard as in the picture below [[Image(http://eden.sahanafoundation.org/raw-attachment/wiki/InstallationGuidelines/Amazon/Dashboard.png, 75%)]] * Next, click on Launch Instance. This new instance will act as a master node for future deployments. We'll use Debian as the OS but other distributions may be supported in the future. In the "Select your Amazon Machine Image" pane, select "AWS Marketplace" from the left sidebar. Scroll down, click on "Operating Systems" and select "Debian GNU/Linux" [[Image(http://eden.sahanafoundation.org/raw-attachment/wiki/InstallationGuidelines/Amazon/SelectAMI.png, 75%)]] * In the next step, select the instance depending on your requirements. * Next, in the Configure Instance pane, expand the Advanced Details tab and enter the contents of https://github.com/gnarula/eden_playbook/blob/master/user-data.sh in the User-Data field. [[Image(http://eden.sahanafoundation.org/raw-attachment/wiki/InstallationGuidelines/Amazon/UserData.png, 75%)]] * Create a new security group with a rule for SSH and HTTP and Port 8000 as shown in the screenshot below. [[Image(http://eden.sahanafoundation.org/raw-attachment/wiki/InstallationGuidelines/Amazon/SecurityGroup.png, 75%)]] * Finally, review the configuration and launch the instance. You'll be prompted to select an existing key pair or generate a new one. In case you're a new user or you don't have access to your key, generate a new keypair, save it and proceed. That's it! The instance will have Deployment Coapp Installed and may be used to install Eden on that instance or any other remote servers. Do note that the installation takes some time (~5 minutes) even after the EC2 Web UI states the instance as "Running". The Coapp may be accessed by browsing to http://instance-public-ip:8000/ Logs for the process may be found by sshing into the newly created instance with your key and viewing /var/log/user-data.log and /var/log/syslog == Next == [wiki:UserGuidelines/Admin Administration Guide]