start tensorflow docker container with jupyter and tensorboard together.

Well, today we’re gonna to have new journey on AWS.

I’d like to have some practices of tensorflow on AWS EC2, to make it simple, I decided to use the tensorflow docker image. (You can install tensorflow on EC2 directly, or just use the AWS deep learning AMI, or build the source code of tensorflow if you like)

This is quite simple actually, just pull the docker image and run it, more detail can be found in tensorflow installation page.

Run below command on EC2 instance:

docker run -d -p 8888:8888 gcr.io/tensorflow/tensorflow

then visit http://<EC2 IP>:8888, you can found jupyter is there waiting for you ūüôā

NB: Like usually, please make sure you have port 8888 of EC2 instance is reachable from public internet. (add inbound rule of your EC2 security group).

When I played a little with tensorflow, I found that there is no tensorboard I can use, but if I hack into the running docker container, and start tensorboard from command line, it’s there !

So, tensorboard is already in the docker image, but not able to use outside. The point is, according to the design of docker, one container should only have one working process to run (we already have jupyter, so…), but we still can make it possible.

No more bothering, It’s also quite simple, just use ‘docker exec’ command to run tensorboard after jupyter is running.

Here is the scirpt

#!/bin/bash
docker run --name tensorflow \
 -d \
 -v /root/tensorflow/notebooks:/notebooks \
 -v /root/tensorflow/tflogs:/tflogs \
 -e "PASSWORD=tensorflow" \
 -p 8888:8888 \
 -p 6006:6006 \
 gcr.io/tensorflow/tensorflow \


sleep 3

docker exec tensorflow nohup tensorboard --logdir /tflogs &

That’s it ! Now you have your tensorboard at http://<EC2 IP>:6066¬†waiting for you.

NB: Like usually, please make sure you have port 6006 of EC2 instance is reachable from public internet. (add inbound rule of your EC2 security group).

Let’s have fun with tensorflow on EC2.

Set up nginx+multiple wordpress+RDS on EC2 with Docker

This is the version 2 of previous post “Set up nginx+wordpress+RDS on EC2 instance with Docker

In this journey, with some tricky, I make the ‘official’ wordpress docker images can support multiple website.
Important: This is definitely NOT the same as wordpress multisite (which is a build-in function of wordpress, but with terrible configuration work. )

My idea just as following:
1.  Nginx (1 instance), this is one docker container.
2. php-fpm (1 instance) + multiple wordpress (m independent instances) , this is another docker container.
3. Mysql (can be local mysql or docker container or AWS RDS or any connectable Mysql), I will chose ASW RDS (as it’s free ūüôā

See below picture for a better view.

Ok, let’s start. I will only record the differences from the previous post¬†“Set up nginx+wordpress+RDS on EC2 instance with Docker“, for more detail, just refer to that post.

1.  Prerequisites

  • You should have EC2 with public or elastic IP assigned, or make it accessible from public internet.
  • You should have your EC2 Security Group enabled 80 (http), 443(https) inbound connection.
  • You should have docker, docker-compose installed on EC2 instance.
  • You should have a connectable Mysql, for my case, I set up a RDS Mysql instance on AWS

2. Configure nginx.

Just like the previous post, you may need to use your own nginx.conf and server.conf files. I won’t paste the full configuration files here, will only list the key differences here.

A: nginx.conf, there is no difference of this file.

B: serverX.conf, the only difference is the php-fpm configuration.

server_name         serverX.com www.serverX.com;
listen              80;
root                /apps/websites/serverX/pages;

location ~ \.php$ {
    root /apps/websites/serverX/pages;
    fastcgi_pass wp:9000;
 
    # other configuration for php-fpm
}
  • NB1: Since will we support multiple servers (or domains/websites), you should have multiple configuration file for each server, like server1.conf, server2.conf, … serverN.conf
  • NB2: /apps/websites/serverX/pages will be the website root path used by wordpress container, Also this path (same path) is used by nginx container.
    So you must mount exactly the same path as /apps/websites/ for both wordpress and nginx containers. You will see the configuration later.
  • NB3: wp is the container name of wordpress (we will go for later)
  • NB4: make sure each serverX.conf has it’s owned servername and root, and all the server will¬†listen on 80 port

C: nginx.yaml

version: '3'
services:
  nginx:
    image: nginx:latest
    volumes:
      #mount the nginx.conf and serverX.conf and the websites root
      - /apps/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - /apps/nginx/conf:/etc/nginx/conf.d:ro
      - /apps/nginx/logs:/var/log/nginx:rw
      - /apps/websites:/apps/websites:ro
    restart: always
    ports:
      - "80:80"
      - "443:443"
    networks:
      - websites
networks:
  websites:
    external:
      name: websites

3. Configure wordpress

Since the official wordpress docker image only support one website, and with fixed default site root at /var/www/html/, so we have to do something to make it more flexible to support ONE php-fpm + Multiple WordPress

3.1 The AS-IS official wordpress docker image behavior

The default ‘entrypoint‘ of¬†official wordpress docker image is: docker-entrypoint.sh,¬†In container it’s under:¬†/usr/local/bin/docker-entrypoint.sh. You can use ‘docker inspect <wordpress image>’ to check (just make sure you already pulled the image from docker hub).

The default docker-entrypoint.sh actually done 3 things:

  • A: Copy wordpress files to ‘/var/www/html’, and update the ‘wp-config.php’ file according to the ‘environment variables’ setting. (like DB_HOST, DB_NAME, DB_ROOT_PASSWORD etc)
  • B: Remove the ‘environment variables’ to make the container more security. Refer to the github doc as following:
    #now that we’re definitely done writing configuration, let’s clear out the relevant envrionment variables (so that stray “phpinfo()” calls don’t leak secrets from our code)
  • C: Start up php-fpm

3.2 Our requirements are:

  • Multiple wordpress installation,
  • single php-fpm running,
  • can be easily add new website (update wordpress) if necessary

3.3 Things to do to meet our requirements

  • build up our own ‘entrypoint’ script, to install multiple wordpress, and start single php-fpm (will not remove the¬†‘environment variables’ , to enable add new website), let’s call it ‘startup.sh‘Here is the startup.sh, it’s quite simple
    #!/bin/bash
    set -euo pipefail
    
    # install/update all the websites under $WEBSITES_ROOT
    installWebsites.sh
    
    # start up php-fpm
    exec php-fpm
  • build up 2 scripts, one will install multiple wordpress (used for all websites full installation or update), let’s call it ‘installWebsites.sh‘, and another will install single wordpress (used by single website install or update), let’s call it ‘installWebsite.sh‘, it will be invoked by ‘InstallWebsites.sh’ normally.¬†Here is the¬†installWebsites.sh

    #!/bin/bash
    set -euo pipefail
    
    # $WEBSITES_ROOT is the root path where all the websites exists in wordpress container
    # in my case, it's /apps/websites, it's an env variable can be set in docker-compose
    cd $WEBSITES_ROOT
    
    # Iterate all the folder (each folder means a separate website root path)
    for d in *
    do
     if [[ -d $d ]]; then
     cd ${d}/pages
     #WORDPRESS_TABLE_PREFIX=${d}_
     # In each folder install wordpress
     # the second argument '${d}_' is used as wordpress table_prefix.
     # normally the the folder name (${d}) is the server's domain name like (google, example, server1 etc)
     installWebsite.sh php-fpm ${d}_
     cd $WEBSITES_ROOT
     fi
    done

    And here is the key part of¬†installWebsite.sh, actually it’s almost same as the default ‘entrypoint’ script¬†docker-entrypoint.sh, copy it, and just comment out the unset ‘env variables’ and start php-fpm parts, and add a little tricky for the table_prefix of each website in wordpress DB.

    #comment out the unset environment variable part
    # now that we're definitely done writing configuration, let's clear out the relevant envrionment variables (so that stray "phpinfo()" calls don't leak secrets from our code)
    #for e in "${envs[@]}"; do
    # unset "$e"
    #done
    
    #comment out the 'starting php-fpm' part
    #exec "$@"
    
    #add table_prefix tricky after the default process
    # this is the original process, find it, and add the tricky part below.
    if [ "$WORDPRESS_TABLE_PREFIX" ]; then
     set_config '$table_prefix' "$WORDPRESS_TABLE_PREFIX"
    fi
    # this is the tricky part, '$2' is the second argument, passed by installWebsites.sh.
    if [ "$2" ]; then
     set_config '$table_prefix' "$2"
    fi
  • change the¬†default¬† ‘entrypoint’ of wordpress container, this is can be done by docker-compose files (or you can use docker run if you like), and mount the scripts we have created to wordpress container.
    Here is the wordpress docker-compose file: servers.yaml
version: '3'

services:
  #the name will be used in nginx serverX.conf
  wp:
    image: wordpress:4.8.2-fpm
    volumes:
       #mount the scripts we build up, also the websites root
       - /apps/websites/startup.sh:/usr/local/bin/startup.sh
       - /apps/websites/installWebsites.sh:/usr/local/bin/installWebsites.sh
       - /apps/websites/installWebsite.sh:/usr/local/bin/installWebsite.sh
       - /apps/websites:/apps/websites
    restart: always
    environment:
       WORDPRESS_DB_HOST: aws-rds-mysql-url-with-port-3306
       WORDPRESS_DB_USER: wordpressuser
       WORDPRESS_DB_PASSWORD: wordpresspass
       #the multiple wordpress websites root, each website with it's own domain name as sub-folder in it.
       WEBSITES_ROOT: /apps/websites
    networks:
      - websites
    #override the default entrypoint of wordpress images
    entrypoint:
      - startup.sh
networks:
  websites:
    external:
      name: websites

4. Prepare websites information.

Let’s start from 2 websites.

Suppose we will setup 2 sites: server1.com and server2.com. Then we need to prepare:

Nginx conf files: server1.conf, server2.conf, refer to the Nginx configure step. Assume you already have the example.conf (the default one we created in previous post), then you can do it as following:

For server1.com

# copy the default one as template
cp example.conf server1.conf

# replace all the domain name from 'example' to 'server1'
sed -i 's/example.com/server1.com/g' server1.conf 
sed -i 's/example/server1/g' server1.conf 


# manually change the other key part mentioned above, e.g root, port, php-fpm config

For server2.com, it’s much easier

# copy the server1.conf as template
cp server1.conf server2.conf

# replace all the domain name from 'server1' to 'server2'
sed -i 's/server1.com/server2.com/g' server2.conf 
sed -i 's/server1/server2/g' server2.conf

Empty website folder for server1.com and sever2.com

mkdir -p /apps/websites/server1/pages
mkdir -p /apps/websites/server2/pages

Ok, now we almost have all set, just a few steps to prepare the nginx serverX.conf for each website, and  start the docker containers, then all will be set.

5. Startup wordpress. (php-fpm + multiple wordpress)

At the same path of servers.yaml (for me, it’s /apps/websites/servers.yaml), run:

docker-compose -f servers.yaml up -d

6. Startup nginx

At the same path of nginx.yaml (for me, it’s /apps/nginx/nginx.yaml), run:

docker-compose -f nginx.yaml up -d

7. Update your website domain DNS setting

Refer to previous post.

8. Add/remove a website.

Let’s say we want add a new website, server3.com. How could I do to achieve this ?

A: Prepare website info
Follow step 4  to make server4.conf (put it under nginx conf folder: /apps/nginx/conf)  and website folder /app/websites/server4/pages.
mkdir -p /app/websites/

B: Install new wordpress.
At the same path of servers.yaml (for me, it’s /apps/websites/servers.yaml), run:

docker-compose -f servers.yaml exec wp installWebsites.sh

C: Reload nginx configuration
At the same path of nginx.yaml (for me, it’s /apps/nginx/nginx.yaml), run:

docker-compose -f nginx.yaml exec nginx service nginx reload

Similar approach when you want to remove website, just remove the nginx serverX.conf, and corresponding website folder (rm -rf /apps/websites/serverX). then run the 2 docker-compose commands (same as add website).

That’s it, we all set.

Hope you can have a good journey with me on AWS.

 

Set up nginx+wordpress+RDS on EC2 instance with Docker

This is my second journey on AWS, at previous post, I practiced to setup LNMP on EC2 with RDS. This time I do it with Docker, which make the journey much easier.

Although EC2  is not the best choice for docker (ECS might be according to AWS) , but at least this is an optional.

Ok, let’s start.

1.¬† Install docker on EC2„Äā

Quite simple if you have to version preference of docker.

sudo yum install docker

2. Download the related docker images.

You will need nginx, wordpress (fpm version), that’s all. (If you want to use mysql locally, you also need mysql docker image)

docker pull nginx #use the latest version is ok

docker pull wordpress:4.8.2-fpm #use the latest fpm version

docker pull mysql #if you want local mysql

3.  Install docker-compose

Docker-compose is optional, but I’d like to use it to run/maintain my container, you can just use ‘docker run’ to do the same work.

Follow the official  document to install docker-compose.

sudo curl -L https://github.com/docker/compose/releases/download/1.16.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose

sudo chmod +x /usr/local/bin/docker-compose

4. Configure nginx.

Just like the previous post, you may need to use your own nginx.conf and server.conf files. I won’t paste the full configuration files here (you can refer to the previous post, ¬†I will only list the key differences here.

A: nginx.conf, there is no difference of this file.

B: server.conf, the only difference is the php-fpm configuration.

root /var/www/html;  #was /apps/websites/example/pages;
#NB:/var/www/html is the default root path used by wordpress container

fastcgi_pass example_wp:9000; #was unix:/var/run/php-fpm/php-fpm.sock
#NB:example_wp is the container name of wordpress (we will go for later)

5. Setup docker network

This step is crucial, we will create a local¬† bridge docker network (named “websites”) to connect all the containers (nginx + wordpress).

docker network create -d bridge websites

NB: you can create the network in docker-compose configuration file, but I don’t like that style, that makes the dependency between different containers.

6. Create AWS RDS for Mysql.

I just copy it from the previous post,¬†maybe I should create a separated post for this part ūüôā

NB: We will not install Mysql on EC2 instance, since it will consume lot of resource which the free tier EC2 instance may not able to afford. Fortunately, AWS is so generous, it provides a separate RDS instance which can be used for Mysql in our case.

This is quite simple too, just follow the document will do the job. Just record following information when you’re done:

Endpoint: The public url which you can access remotely.

DB instance name: like wordpress

DB master username (not root): like wordpress_user

DB master user password:

Another thing need your attention is:

Make sure the ‚ÄėSecurity Group‚Äô¬† you used with following inbound rule:

MYSQL/Aurora
TCP
3306
<the subnet where your want to access the DB>
or 0.0.0.0/0 for public access.

7. Startup wordpress.

Create a example.yaml docker-compose file like following:

version: '3'

services:
   example_wp:
     image: wordpress:4.8.2-fpm
     volumes:
       - /apps/websites/example/pages:/var/www/html
     restart: always
     environment:
       WORDPRESS_DB_HOST: aws-rds-mysql-url-with-port-3306
       WORDPRESS_DB_USER: wordpressuser
       WORDPRESS_DB_PASSWORD: wordpresspass
     networks:
      - websites

networks:
  websites:
    external:
      name: websites

Some remarks:
A:¬† The bind volume “/apps/websites/example/pages” is the place where you want to install the ‘wordpress’ (will done by the wordpress container), if you don’t want to modify the ‘wordpress’ installation/configuration, you do not need the ‘volume’ setting here.

B: The environment variables, just put your AWS RDS information which you created in step 6  here is ok.

C: The network, make sure you use the same network name which created in step 5.

Then you can startup the workpress container as following:

docker-compose -f example.yaml up -d

8. Startup nginx

Create nginx.yaml file like following:

version: '3'
services:
  nginx:
    image: nginx:latest
    volumes:
      - /apps/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - /apps/nginx/conf:/etc/nginx/conf.d:ro
      - /apps/nginx/logs:/var/log/nginx:rw
      - /apps/websites:/apps/websites:ro
    restart: always
    ports:
      - "80:80"
      - "443:443"
    networks:
      - websites
networks:
  websites:
    external:
      name: websites

Also some remarks:
A: The bind volumes, export the nginx configuration files (nginx.conf + server.conf), just in case we can simple update them, export nginx log for easy problem analyzing, export the website root path, so we can make sure ‘nginx’ and ‘wordpress’ use the same website files.

Another thing need your attention is: only log file was bond as ‘rw’ (read-write) mode, other files are ‘ro’ (read-only) mode, that prevents the container change the file by accident (you will never know what have done in the container if you’re not the owner of it ūüôā

B: The port,¬†this is the ports nginx listening for http/https request, you have to make sure you setup corresponding ‘in bound’ rules(open 80/443 port http/https connection request for any IP [0.0.0.0/0]) in your EC2 Security Group. (something like below)

HTTP
TCP
80
0.0.0.0/0
HTTPS
TCP
443
0.0.0.0/0

C: The network, make sure you use the same network name which created in step 5.

Then you can startup the nginx container as following:

docker-compose -f nginx.yaml up -d

9. Finally and optionally,¬†update your website domain DNS setting„Äā

If you just want a test/develop environment, this step could be skip, update your domain DNS setting in you domain host (like godaddy), point your domain name to the public or Elastic IP of your EC2 instance.

I think that’s all. You should be able to visit your website via:

http://www.example.com (just in case you did the last step)

http://<public or elastic IP of EC2 instance>

Hope you can have a good journey with me on AWS.