{"_id":"550556cdc16b21170080c646","project":"5503ea178c5e913700362c70","__v":74,"version":{"_id":"55053eeb84ad8c0d005b0a62","__v":2,"forked_from":"5503ea188c5e913700362c73","project":"5503ea178c5e913700362c70","createdAt":"2015-03-15T08:12:27.786Z","releaseDate":"2015-03-15T08:12:27.786Z","categories":["55053eec84ad8c0d005b0a63","550556a4728deb23005ec0f0"],"is_deprecated":false,"is_hidden":false,"is_beta":true,"is_stable":false,"codename":"","version_clean":"0.0.5","version":"0.0.5"},"category":{"_id":"550556a4728deb23005ec0f0","__v":6,"pages":["550556cdc16b21170080c646","55055707b9a7a0190036697c","5507638ffa89210d00c8c987","5507ebef6ac1620d001b9405","5508995e0f146f3500b031ae","552ada773f29c30d00619cbc"],"version":"55053eeb84ad8c0d005b0a62","project":"5503ea178c5e913700362c70","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2015-03-15T09:53:40.258Z","from_sync":false,"order":1,"slug":"tutorials","title":"Tutorials"},"user":"5503e897e508a017002013bd","githubsync":"","metadata":{"title":"","description":"","image":[]},"updates":[],"next":{"pages":[],"description":""},"createdAt":"2015-03-15T09:54:21.584Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"auth":"required","params":[],"url":""},"isReference":false,"order":1,"body":"If you followed [Tutorial: First Steps](doc:tutorial-first-steps), you learned how to train and use an existing model on a dataset. In this tutorial, you will see how easy it is to implement your own models in OpenDeep.\n\nMost other places will have you start with a traditional feed-forward Multilayer Perceptron, but we know you are better than that! Let's implement something cool like an unsupervised denoising autoencoder and use it to reconstruct MNIST. Then in another tutorial will show you how to make it into a supervised classifier.\n\nBefore you begin, you should be familiar with [Theano](http://deeplearning.net/software/theano/), especially with the basics in [baby steps - algebra](http://deeplearning.net/software/theano/tutorial/adding.html) for creating variables and functions. It is also a good idea to set up [Theano to use the GPU](http://deeplearning.net/software/theano/tutorial/using_gpu.html) to reduce training time.\n\nI also recommend knowing the basics of how autoencoders work. There is already a great tutorial in Theano for denoising autoencoders here: [Denoising Autoencoder](http://deeplearning.net/tutorial/dA.html).\n\nIn this tutorial, we are going to start with the bare minimum for creating a denoising autoencoder to work with MNIST data, and then show how to make it flexible in the OpenDeep style!\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Creating a bare-bones Model\"\n}\n[/block]\nThe Model class in OpenDeep provides an interface for the functionality a model needs to work with Datasets and Optimizers. At a minimum to be useful, a model needs to be able to train and predict. This means you only need to write four methods. Yes, you heard right, only four methods:\n* get_inputs(): return a list of the inputs your model accepts. In our case, a single matrix.\n* get_params(): return a list of the model parameters. In our case, a weights matrix and a bias vector.\n* get_train_cost(): return an expression of the training cost for the model.\n* get_outputs(): return a Theano expression for computing outputs given inputs - the whole point of having a model!\n\nLet's jump right in!\n#Initializing a denoising autoencoder\nDuring initialization, you can set up the hyperparameters and perform the main computations for your model (not best practice to be monolithic like this, but you can get away with it for personal research).\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"import theano.tensor as T\\nfrom opendeep.models.model import Model\\nfrom opendeep.utils.nnet import get_weights_uniform, get_bias\\nfrom opendeep.utils.noise import salt_and_pepper\\nfrom opendeep.utils.activation import tanh, sigmoid\\nfrom opendeep.utils.cost import binary_crossentropy\\n\\n# create our class initialization!\\nclass DenoisingAutoencoder(Model):\\n    \\\"\\\"\\\"\\n    A denoising autoencoder will corrupt an input (add noise) and try to reconstruct it.\\n    \\\"\\\"\\\"\\n    def __init__(self):\\n        # Define some model hyperparameters to work with MNIST images!\\n        input_size  = 28*28 # dimensions of image\\n        hidden_size = 1000  # number of hidden units - generally bigger than input size for DAE\\n\\n        # Now, define the symbolic input to the model (Theano)\\n        # We use a matrix rather than a vector so that minibatch processing can be done in parallel.\\n        x = T.fmatrix(\\\"X\\\")\\n        self.inputs = [x]\\n\\n        # Build the model's parameters - a weight matrix and two bias vectors\\n        W  = get_weights_uniform(shape=(input_size, hidden_size), name=\\\"W\\\")\\n        b0 = get_bias(shape=input_size, name=\\\"b0\\\")\\n        b1 = get_bias(shape=hidden_size, name=\\\"b1\\\")\\n        self.params = [W, b0, b1]\\n\\n        # Perform the computation for a denoising autoencoder!\\n        # first, add noise (corrupt) the input\\n        corrupted_input = salt_and_pepper(input=x, corruption_level=0.4)\\n        # next, compute the hidden layer given the inputs (the encoding function)\\n        hiddens = tanh(T.dot(corrupted_input, W) + b1)\\n        # finally, create the reconstruction from the hidden layer (we tie the weights with W.T)\\n        reconstruction = sigmoid(T.dot(hiddens, W.T) + b0)\\n        # the training cost is reconstruction error - with MNIST this is binary cross-entropy\\n        self.train_cost = binary_crossentropy(output=reconstruction, target=x)\\n\\n        # Compile everything into a Theano function for prediction!\\n        # When using real-world data in predictions, we wouldn't corrupt the input first.\\n        # Therefore, create another version of the hiddens and reconstruction without adding the noise\\n        hiddens_predict = tanh(T.dot(x, W) + b1)\\n        recon_predict   = sigmoid(T.dot(hiddens_predict, W.T) + b0)\\n        self.output     = recon_predict\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\nThen all we need to do is add the four methods listed above. Because we created the model's computational logic in the `__init__` method, this is as simple as returning the variables. See below:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"def get_inputs(self):\\n    return self.inputs\\n\\ndef get_params(self):\\n    return self.params\\n\\ndef get_train_cost(self):\\n    return self.train_cost\\n\\ndef get_outputs(self):\\n    return self.output\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\nThat was easy! Now, we can train and evaluate our shiny new model. To put everything together, here is the completed class with code to run on MNIST:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"import theano.tensor as T\\nfrom opendeep.models.model import Model\\nfrom opendeep.utils.nnet import get_weights_uniform, get_bias\\nfrom opendeep.utils.noise import salt_and_pepper\\nfrom opendeep.utils.activation import tanh, sigmoid\\nfrom opendeep.utils.cost import binary_crossentropy\\n\\n# create our class initialization!\\nclass DenoisingAutoencoder(Model):\\n    \\\"\\\"\\\"\\n    A denoising autoencoder will corrupt an input (add noise) and try to reconstruct it.\\n    \\\"\\\"\\\"\\n    def __init__(self):\\n        # Define some model hyperparameters to work with MNIST images!\\n        input_size  = 28*28 # dimensions of image\\n        hidden_size = 1000  # number of hidden units - generally bigger than input size for DAE\\n\\n        # Now, define the symbolic input to the model (Theano)\\n        # We use a matrix rather than a vector so that minibatch processing can be done in parallel.\\n        x = T.fmatrix(\\\"X\\\")\\n        self.inputs = [x]\\n\\n        # Build the model's parameters - a weight matrix and two bias vectors\\n        W  = get_weights_uniform(shape=(input_size, hidden_size), name=\\\"W\\\")\\n        b0 = get_bias(shape=input_size, name=\\\"b0\\\")\\n        b1 = get_bias(shape=hidden_size, name=\\\"b1\\\")\\n        self.params = [W, b0, b1]\\n\\n        # Perform the computation for a denoising autoencoder!\\n        # first, add noise (corrupt) the input\\n        corrupted_input = salt_and_pepper(input=x, corruption_level=0.4)\\n        # next, compute the hidden layer given the inputs (the encoding function)\\n        hiddens = tanh(T.dot(corrupted_input, W) + b1)\\n        # finally, create the reconstruction from the hidden layer (we tie the weights with W.T)\\n        reconstruction = sigmoid(T.dot(hiddens, W.T) + b0)\\n        # the training cost is reconstruction error - with MNIST this is binary cross-entropy\\n        self.train_cost = binary_crossentropy(output=reconstruction, target=x)\\n\\n        # Compile everything into a Theano function for prediction!\\n        # When using real-world data in predictions, we wouldn't corrupt the input first.\\n        # Therefore, create another version of the hiddens and reconstruction without adding the noise\\n        hiddens_predict = tanh(T.dot(x, W) + b1)\\n        recon_predict   = sigmoid(T.dot(hiddens_predict, W.T) + b0)\\n        self.output\\t\\t  = recon_predict\\n\\n    def get_inputs(self):\\n        return self.inputs\\n\\n    def get_params(self):\\n        return self.params\\n\\n    def get_train_cost(self):\\n        return self.train_cost\\n\\n    def get_outputs(self):\\n        return self.output\\n\\nif __name__ == '__main__':\\n    # set up the logging environment to display outputs (optional)\\n    # although this is recommended over print statements everywhere\\n    import logging\\n    from opendeep.log.logger import config_root_logger\\n    config_root_logger()\\n    log = logging.getLogger(__name__)\\n    log.info(\\\"Creating a Denoising Autoencoder!\\\")\\n\\n    # import the dataset and optimizer to use\\n    from opendeep.data.dataset import TEST\\n    from opendeep.data.standard_datasets.image.mnist import MNIST\\n    from opendeep.optimization.adadelta import AdaDelta\\n\\n    # grab the MNIST dataset\\n    mnist = MNIST()\\n\\n    # create your shiny new DAE\\n    dae = DenoisingAutoencoder()\\n\\n    # make an optimizer to train it (AdaDelta is a good default)\\n    optimizer = AdaDelta(model=dae, dataset=mnist)\\n    # perform training!\\n    optimizer.train()\\n\\n    # test it on some images!\\n    test_data = mnist.getDataByIndices(indices=range(25), subset=TEST)\\n    corrupted_test = salt_and_pepper(test_data, 0.4).eval()\\n    # use the predict function!\\n    reconstructed_images = dae.predict(corrupted_test)\\n\\n    # create an image from this reconstruction!\\n    # imports for working with tiling outputs into one image\\n    from opendeep.utils.image import tile_raster_images\\n    import numpy\\n    import PIL\\n    # stack the image matrices together in three 5x5 grids next to each other using numpy\\n    stacked = numpy.vstack(\\n        [numpy.vstack([test_data[i*5 : (i+1)*5],\\n                       corrupted_test[i*5 : (i+1)*5],\\n                       reconstructed_images[i*5 : (i+1)*5]])\\n         for i in range(5)])\\n    # convert the combined matrix into an image\\n    image = PIL.Image.fromarray(\\n        tile_raster_images(stacked, (28, 28), (5, 3*5))\\n    )\\n    # save it!\\n    image.save(\\\"dae_reconstruction_test.png\\\")\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\nHuzzah! This code should produce an image that looks like this after ~350 training epochs:\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/OkJ3eJFkQJm0cWnGxyIe_dae_reconstruction_test.png\",\n        \"dae_reconstruction_test.png\",\n        \"420\",\n        \"140\",\n        \"\",\n        \"\"\n      ],\n      \"caption\": \"Left: original test images.\\nCenter: corrupted (noisy) images.\\nRight: reconstructed images (output).\"\n    }\n  ]\n}\n[/block]\nCongrats, you just created a brand new unsupervised model!\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/HmdDoxuIQoCx884Cnpwi_its_working_star_wars.gif\",\n        \"its_working_star_wars.gif\",\n        \"400\",\n        \"170\",\n        \"\",\n        \"\"\n      ]\n    }\n  ]\n}\n[/block]","excerpt":"Write your own denoising autoencoder and train it on MNIST","slug":"tutorial-your-first-model","type":"basic","title":"Tutorial: Your First Model (DAE)"}

Tutorial: Your First Model (DAE)

Write your own denoising autoencoder and train it on MNIST

If you followed [Tutorial: First Steps](doc:tutorial-first-steps), you learned how to train and use an existing model on a dataset. In this tutorial, you will see how easy it is to implement your own models in OpenDeep. Most other places will have you start with a traditional feed-forward Multilayer Perceptron, but we know you are better than that! Let's implement something cool like an unsupervised denoising autoencoder and use it to reconstruct MNIST. Then in another tutorial will show you how to make it into a supervised classifier. Before you begin, you should be familiar with [Theano](http://deeplearning.net/software/theano/), especially with the basics in [baby steps - algebra](http://deeplearning.net/software/theano/tutorial/adding.html) for creating variables and functions. It is also a good idea to set up [Theano to use the GPU](http://deeplearning.net/software/theano/tutorial/using_gpu.html) to reduce training time. I also recommend knowing the basics of how autoencoders work. There is already a great tutorial in Theano for denoising autoencoders here: [Denoising Autoencoder](http://deeplearning.net/tutorial/dA.html). In this tutorial, we are going to start with the bare minimum for creating a denoising autoencoder to work with MNIST data, and then show how to make it flexible in the OpenDeep style! [block:api-header] { "type": "basic", "title": "Creating a bare-bones Model" } [/block] The Model class in OpenDeep provides an interface for the functionality a model needs to work with Datasets and Optimizers. At a minimum to be useful, a model needs to be able to train and predict. This means you only need to write four methods. Yes, you heard right, only four methods: * get_inputs(): return a list of the inputs your model accepts. In our case, a single matrix. * get_params(): return a list of the model parameters. In our case, a weights matrix and a bias vector. * get_train_cost(): return an expression of the training cost for the model. * get_outputs(): return a Theano expression for computing outputs given inputs - the whole point of having a model! Let's jump right in! #Initializing a denoising autoencoder During initialization, you can set up the hyperparameters and perform the main computations for your model (not best practice to be monolithic like this, but you can get away with it for personal research). [block:code] { "codes": [ { "code": "import theano.tensor as T\nfrom opendeep.models.model import Model\nfrom opendeep.utils.nnet import get_weights_uniform, get_bias\nfrom opendeep.utils.noise import salt_and_pepper\nfrom opendeep.utils.activation import tanh, sigmoid\nfrom opendeep.utils.cost import binary_crossentropy\n\n# create our class initialization!\nclass DenoisingAutoencoder(Model):\n \"\"\"\n A denoising autoencoder will corrupt an input (add noise) and try to reconstruct it.\n \"\"\"\n def __init__(self):\n # Define some model hyperparameters to work with MNIST images!\n input_size = 28*28 # dimensions of image\n hidden_size = 1000 # number of hidden units - generally bigger than input size for DAE\n\n # Now, define the symbolic input to the model (Theano)\n # We use a matrix rather than a vector so that minibatch processing can be done in parallel.\n x = T.fmatrix(\"X\")\n self.inputs = [x]\n\n # Build the model's parameters - a weight matrix and two bias vectors\n W = get_weights_uniform(shape=(input_size, hidden_size), name=\"W\")\n b0 = get_bias(shape=input_size, name=\"b0\")\n b1 = get_bias(shape=hidden_size, name=\"b1\")\n self.params = [W, b0, b1]\n\n # Perform the computation for a denoising autoencoder!\n # first, add noise (corrupt) the input\n corrupted_input = salt_and_pepper(input=x, corruption_level=0.4)\n # next, compute the hidden layer given the inputs (the encoding function)\n hiddens = tanh(T.dot(corrupted_input, W) + b1)\n # finally, create the reconstruction from the hidden layer (we tie the weights with W.T)\n reconstruction = sigmoid(T.dot(hiddens, W.T) + b0)\n # the training cost is reconstruction error - with MNIST this is binary cross-entropy\n self.train_cost = binary_crossentropy(output=reconstruction, target=x)\n\n # Compile everything into a Theano function for prediction!\n # When using real-world data in predictions, we wouldn't corrupt the input first.\n # Therefore, create another version of the hiddens and reconstruction without adding the noise\n hiddens_predict = tanh(T.dot(x, W) + b1)\n recon_predict = sigmoid(T.dot(hiddens_predict, W.T) + b0)\n self.output = recon_predict", "language": "python" } ] } [/block] Then all we need to do is add the four methods listed above. Because we created the model's computational logic in the `__init__` method, this is as simple as returning the variables. See below: [block:code] { "codes": [ { "code": "def get_inputs(self):\n return self.inputs\n\ndef get_params(self):\n return self.params\n\ndef get_train_cost(self):\n return self.train_cost\n\ndef get_outputs(self):\n return self.output", "language": "python" } ] } [/block] That was easy! Now, we can train and evaluate our shiny new model. To put everything together, here is the completed class with code to run on MNIST: [block:code] { "codes": [ { "code": "import theano.tensor as T\nfrom opendeep.models.model import Model\nfrom opendeep.utils.nnet import get_weights_uniform, get_bias\nfrom opendeep.utils.noise import salt_and_pepper\nfrom opendeep.utils.activation import tanh, sigmoid\nfrom opendeep.utils.cost import binary_crossentropy\n\n# create our class initialization!\nclass DenoisingAutoencoder(Model):\n \"\"\"\n A denoising autoencoder will corrupt an input (add noise) and try to reconstruct it.\n \"\"\"\n def __init__(self):\n # Define some model hyperparameters to work with MNIST images!\n input_size = 28*28 # dimensions of image\n hidden_size = 1000 # number of hidden units - generally bigger than input size for DAE\n\n # Now, define the symbolic input to the model (Theano)\n # We use a matrix rather than a vector so that minibatch processing can be done in parallel.\n x = T.fmatrix(\"X\")\n self.inputs = [x]\n\n # Build the model's parameters - a weight matrix and two bias vectors\n W = get_weights_uniform(shape=(input_size, hidden_size), name=\"W\")\n b0 = get_bias(shape=input_size, name=\"b0\")\n b1 = get_bias(shape=hidden_size, name=\"b1\")\n self.params = [W, b0, b1]\n\n # Perform the computation for a denoising autoencoder!\n # first, add noise (corrupt) the input\n corrupted_input = salt_and_pepper(input=x, corruption_level=0.4)\n # next, compute the hidden layer given the inputs (the encoding function)\n hiddens = tanh(T.dot(corrupted_input, W) + b1)\n # finally, create the reconstruction from the hidden layer (we tie the weights with W.T)\n reconstruction = sigmoid(T.dot(hiddens, W.T) + b0)\n # the training cost is reconstruction error - with MNIST this is binary cross-entropy\n self.train_cost = binary_crossentropy(output=reconstruction, target=x)\n\n # Compile everything into a Theano function for prediction!\n # When using real-world data in predictions, we wouldn't corrupt the input first.\n # Therefore, create another version of the hiddens and reconstruction without adding the noise\n hiddens_predict = tanh(T.dot(x, W) + b1)\n recon_predict = sigmoid(T.dot(hiddens_predict, W.T) + b0)\n self.output\t\t = recon_predict\n\n def get_inputs(self):\n return self.inputs\n\n def get_params(self):\n return self.params\n\n def get_train_cost(self):\n return self.train_cost\n\n def get_outputs(self):\n return self.output\n\nif __name__ == '__main__':\n # set up the logging environment to display outputs (optional)\n # although this is recommended over print statements everywhere\n import logging\n from opendeep.log.logger import config_root_logger\n config_root_logger()\n log = logging.getLogger(__name__)\n log.info(\"Creating a Denoising Autoencoder!\")\n\n # import the dataset and optimizer to use\n from opendeep.data.dataset import TEST\n from opendeep.data.standard_datasets.image.mnist import MNIST\n from opendeep.optimization.adadelta import AdaDelta\n\n # grab the MNIST dataset\n mnist = MNIST()\n\n # create your shiny new DAE\n dae = DenoisingAutoencoder()\n\n # make an optimizer to train it (AdaDelta is a good default)\n optimizer = AdaDelta(model=dae, dataset=mnist)\n # perform training!\n optimizer.train()\n\n # test it on some images!\n test_data = mnist.getDataByIndices(indices=range(25), subset=TEST)\n corrupted_test = salt_and_pepper(test_data, 0.4).eval()\n # use the predict function!\n reconstructed_images = dae.predict(corrupted_test)\n\n # create an image from this reconstruction!\n # imports for working with tiling outputs into one image\n from opendeep.utils.image import tile_raster_images\n import numpy\n import PIL\n # stack the image matrices together in three 5x5 grids next to each other using numpy\n stacked = numpy.vstack(\n [numpy.vstack([test_data[i*5 : (i+1)*5],\n corrupted_test[i*5 : (i+1)*5],\n reconstructed_images[i*5 : (i+1)*5]])\n for i in range(5)])\n # convert the combined matrix into an image\n image = PIL.Image.fromarray(\n tile_raster_images(stacked, (28, 28), (5, 3*5))\n )\n # save it!\n image.save(\"dae_reconstruction_test.png\")", "language": "python" } ] } [/block] Huzzah! This code should produce an image that looks like this after ~350 training epochs: [block:image] { "images": [ { "image": [ "https://files.readme.io/OkJ3eJFkQJm0cWnGxyIe_dae_reconstruction_test.png", "dae_reconstruction_test.png", "420", "140", "", "" ], "caption": "Left: original test images.\nCenter: corrupted (noisy) images.\nRight: reconstructed images (output)." } ] } [/block] Congrats, you just created a brand new unsupervised model! [block:image] { "images": [ { "image": [ "https://files.readme.io/HmdDoxuIQoCx884Cnpwi_its_working_star_wars.gif", "its_working_star_wars.gif", "400", "170", "", "" ] } ] } [/block]