{"_id":"5507ebef6ac1620d001b9405","user":"5503e897e508a017002013bd","version":{"_id":"55053eeb84ad8c0d005b0a62","__v":2,"forked_from":"5503ea188c5e913700362c73","project":"5503ea178c5e913700362c70","createdAt":"2015-03-15T08:12:27.786Z","releaseDate":"2015-03-15T08:12:27.786Z","categories":["55053eec84ad8c0d005b0a63","550556a4728deb23005ec0f0"],"is_deprecated":false,"is_hidden":false,"is_beta":true,"is_stable":false,"codename":"","version_clean":"0.0.5","version":"0.0.5"},"githubsync":"","project":"5503ea178c5e913700362c70","category":{"_id":"550556a4728deb23005ec0f0","__v":6,"pages":["550556cdc16b21170080c646","55055707b9a7a0190036697c","5507638ffa89210d00c8c987","5507ebef6ac1620d001b9405","5508995e0f146f3500b031ae","552ada773f29c30d00619cbc"],"version":"55053eeb84ad8c0d005b0a62","project":"5503ea178c5e913700362c70","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2015-03-15T09:53:40.258Z","from_sync":false,"order":1,"slug":"tutorials","title":"Tutorials"},"__v":36,"metadata":{"title":"","description":"","image":[]},"updates":[],"next":{"pages":[],"description":""},"createdAt":"2015-03-17T08:55:11.922Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"auth":"required","params":[],"url":""},"isReference":false,"order":999,"body":"This tutorial builds on the Denoising Autoencoder built from [Tutorial: Your First Model (DAE)](doc:tutorial-your-first-model).\n\nOur naïve implementation of the denoising autoencoder works great for personal use on the MNIST dataset, but it isn't very practical to have a model for one dataset. In this tutorial, you will see how to make a model flexible via configurations, and to deal with hooks for linking models together.\n\n#Configurations and defaults\nWith OpenDeep, it is good practice to be able to pass model hyperparameters to the class in a few ways:\n* a JSON file that will be parsed into a dictionary\n* a YAML file that will be parsed into a dictionary\n* a dictionary-like object directly (which means it has a `.get()` method)\n\nIt would also be good to have a \"default\" configuration with the ability to override via a \"config\", or via parameters directly passed into the model. \n\nLuckily there is some code to make this really easy! The `__init__` method of the `Model` class handles parsing the \"default\" and \"config\" dictionaries as well as the explicitly passed parameters into one variable - `self.args`. You can access any configuration parameters from this variable as well as directly from the `self` reference. Here is an example modifying our class's `__init__` from before to handle configurations of hyperparameters:\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"import theano.tensor as T\\nfrom opendeep.models.model import Model\\nfrom opendeep.utils.nnet import get_weights_uniform, get_bias\\nfrom opendeep.utils.noise import salt_and_pepper\\nfrom opendeep.utils.activation import get_activation_function\\nfrom opendeep.utils.cost import get_cost_function\\n\\n# create our class initialization!\\nclass DenoisingAutoencoder(Model):\\n    \\\"\\\"\\\"\\n    A denoising autoencoder will corrupt an input (add noise) and try to reconstruct it.\\n    \\\"\\\"\\\"\\n    # Best practice to define all the default parameters up here\\n    # Provide comments when giving parameters so people know what they are for!\\n    _defaults = {\\n        \\\"input_size\\\": 28*28, # dimensionality of input - works for MNIST\\n        \\\"hidden_size\\\": 1000, # number of hidden units\\n        \\\"corruption_level\\\": 0.4, # how much noise to add to the input\\n        \\\"hidden_activation\\\": 'tanh', # the nonlinearity to apply to hidden units\\n        \\\"visible_activation\\\": 'sigmoid', # the nonlinearity to apply to visible units\\n        \\\"cost_function\\\": 'binary_crossentropy' # the cost function to use during training\\n    }\\n    def __init__(self, config=None, defaults=_defaults, input_size=None, hidden_size=None, corruption_level=None, hidden_activation=None, visible_activation=None, cost_function=None):\\n        # Now, initialize with Model class with all of the initialization variables passed to the denoising autoencoder, except for the self reference.\\n        # The parameters will be combined by overriding each other in this order:\\n        # defaults < config < explicitly passed parameters\\n        super(DenoisingAutoencoder, self).__init__(\\n            **{arg: val for (arg, val) in locals().iteritems() if arg is not 'self'}\\n        )\\n        # These parameters are now accessible from the 'self.args' dictionary, as well as directly accessing from the 'self' reference!\\n\\n        # use the helper methods to grab appropriate activation functions from names!\\n        hidden_activation  = get_activation_function(self.hidden_activation)\\n        visible_activation = get_activation_function(self.visible_activation)\\n\\n        # do the same for the cost function\\n        cost_function  = get_cost_function(self.cost_function)\\n\\n        # Now, define the symbolic input to the model (Theano)\\n        # We use a matrix rather than a vector so that minibatch processing can be done in parallel.\\n        x = T.fmatrix(\\\"X\\\")\\n        self.inputs = [x]\\n\\n        # Build the model's parameters - a weight matrix and two bias vectors\\n        W  = get_weights_uniform(shape=(self.input_size, self.hidden_size), name=\\\"W\\\")\\n        b0 = get_bias(shape=self.input_size, name=\\\"b0\\\")\\n        b1 = get_bias(shape=self.hidden_size, name=\\\"b1\\\")\\n        self.params = [W, b0, b1]\\n\\n        # Perform the computation for a denoising autoencoder!\\n        # first, add noise (corrupt) the input\\n        corrupted_input = salt_and_pepper(input=x, corruption_level=self.corruption_level)\\n        # next, compute the hidden layer given the inputs (the encoding function)\\n        hiddens = hidden_activation(T.dot(corrupted_input, W) + b1)\\n        # finally, create the reconstruction from the hidden layer (we tie the weights with W.T)\\n        reconstruction = visible_activation(T.dot(hiddens, W.T) + b0)\\n        # the training cost is reconstruction error\\n        self.train_cost = cost_function(output=reconstruction, target=x)\\n\\n        # Compile everything into a Theano function for prediction!\\n        # When using real-world data in predictions, we wouldn't corrupt the input first.\\n        # Therefore, create another version of the hiddens and reconstruction without adding the noise\\n        hiddens_predict = hidden_activation(T.dot(x, W) + b1)\\n        recon_predict   = visible_activation(T.dot(hiddens_predict, W.T) + b0)\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\nNot too bad! You just made the model infinitely more customizable!\n\n#Modularity with other models via hooks\nFinally, a key concept in OpenDeep is maintaining modularity between models. For this purpose, there are three inputs a model should handle when applicable:\n* **inputs_hook**: this is a tuple of (size, variable) for an input to your model. It should replace the default input you have set up (in our case, the matrix \"X\").\n* **hiddens_hook**: this is also a tuple of (size, variable) for the hidden representation of your model. It only makes sense in the case of generative models (like denoising autoencoders) where you can compute an output given the hiddens. Therefore hiddens_hook is optional most of the time (unless you have a generative model). This hook is up to your discretion for how the model should handle taking a hidden representation as its input.\n* **params_hook**: a list of variables to replace the default parameters used by your model. This enables models to be 'tied' together by using the same parameters. In our case, we should handle replacing \"W\", \"b0\", and \"b1\".\n\nIt is easier to explain via example with comments:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"import theano.tensor as T\\nfrom opendeep import function\\nfrom opendeep.models.model import Model\\nfrom opendeep.utils.nnet import get_weights_uniform, get_bias\\nfrom opendeep.utils.noise import salt_and_pepper\\nfrom opendeep.utils.activation import get_activation_function\\nfrom opendeep.utils.cost import get_cost_function\\n\\n# create our class initialization!\\nclass DenoisingAutoencoder(Model):\\n    \\\"\\\"\\\"\\n    A denoising autoencoder will corrupt an input (add noise) and try to reconstruct it.\\n    \\\"\\\"\\\"\\n    # Best practice to define all the default parameters up here\\n    # Provide comments when giving parameters so people know what they are for!\\n    _defaults = {\\n        \\\"input_size\\\": 28*28,  # dimensionality of input - works for MNIST\\n        \\\"hidden_size\\\": 1000,  # number of hidden units\\n        \\\"corruption_level\\\": 0.4,  # how much noise to add to the input\\n        \\\"hidden_activation\\\": 'tanh',  # the nonlinearity to apply to hidden units\\n        \\\"visible_activation\\\": 'sigmoid',  # the nonlinearity to apply to visible units\\n        \\\"cost_function\\\": 'binary_crossentropy'  # the cost function to use during training\\n    }\\n    def __init__(self, config=None, defaults=_defaults,\\n                 inputs_hook=None, hiddens_hook=None, params_hook=None,\\n                 input_size=None, hidden_size=None, corruption_level=None,\\n                 hidden_activation=None, visible_activation=None, cost_function=None):\\n        # Now, initialize with Model class to combine config and defaults!\\n        # Here, defaults is defined via a dictionary. However, you could also\\n        # pass a filename to a JSON or YAML file with the same format.\\n        super(DenoisingAutoencoder, self).__init__(\\n            **{arg: val for (arg, val) in locals().iteritems() if arg is not 'self'}\\n        )\\n        # Any parameter from the 'config' will overwrite the 'defaults' dictionary, which will be overwritten if the\\n        # parameter is passed directly to __init__.\\n        # These parameters are now accessible from the 'self' variable!\\n\\n        # Define model hyperparameters\\n        # deal with the inputs_hook and hiddens_hook for the size parameters!\\n        # if the hook exists, grab the size from the first element of the tuple.\\n        if self.inputs_hook is not None:\\n            assert len(self.inputs_hook) == 2, \\\"Was expecting inputs_hook to be a tuple.\\\"\\n            self.input_size = inputs_hook[0]\\n\\n        if self.hiddens_hook is not None:\\n            assert len(self.hiddens_hook) == 2, \\\"was expecting hiddens_hook to be a tuple.\\\"\\n            self.hidden_size = hiddens_hook[0]\\n\\n\\n        # use the helper methods to grab appropriate activation functions from names!\\n        hidden_activation  = get_activation_function(self.hidden_activation)\\n        visible_activation = get_activation_function(self.visible_activation)\\n\\n        # do the same for the cost function\\n        cost_function = get_cost_function(self.cost_function)\\n\\n        # Now, define the symbolic input to the model (Theano)\\n        # We use a matrix rather than a vector so that minibatch processing can be done in parallel.\\n        # Make sure to deal with 'inputs_hook' if it exists!\\n        if self.inputs_hook is not None:\\n            # grab the new input variable from the inputs_hook tuple\\n            x = self.inputs_hook[1]\\n        else:\\n            x = T.fmatrix(\\\"X\\\")\\n        self.inputs = [x]\\n\\n        # Build the model's parameters - a weight matrix and two bias vectors\\n        # Make sure to deal with 'params_hook' if it exists!\\n        if self.params_hook:\\n            # check to see if it contains the three necessary variables\\n            assert len(self.params_hook) == 3, \\\"Not correct number of params to DAE, needs 3!\\\"\\n            W, b0, b1 = self.params_hook\\n        else:\\n            W  = get_weights_uniform(shape=(self.input_size, self.hidden_size), name=\\\"W\\\")\\n            b0 = get_bias(shape=self.input_size, name=\\\"b0\\\")\\n            b1 = get_bias(shape=self.hidden_size, name=\\\"b1\\\")\\n        self.params = [W, b0, b1]\\n\\n        # Perform the computation for a denoising autoencoder!\\n        # first, add noise (corrupt) the input\\n        corrupted_input = salt_and_pepper(input=x, corruption_level=self.corruption_level)\\n        # next, compute the hidden layer given the inputs (the encoding function)\\n        # We don't need to worry about hiddens_hook during training, because we can't\\n        # compute a cost without having the input!\\n        # hiddens_hook is more for the predict function and linking methods below.\\n        hiddens = hidden_activation(T.dot(corrupted_input, W) + b1)\\n        # finally, create the reconstruction from the hidden layer (we tie the weights with W.T)\\n        reconstruction = visible_activation(T.dot(hiddens, W.T) + b0)\\n        # the training cost is reconstruction error\\n        self.train_cost = cost_function(output=reconstruction, target=x)\\n\\n        # Compile everything into a Theano function for prediction!\\n        # When using real-world data in predictions, we wouldn't corrupt the input first.\\n        # Therefore, create another version of the hiddens and reconstruction without adding the noise.\\n        # Here is where we would handle hiddens_hook because this is a generative model!\\n        # For the predict function, it would take in the hiddens instead of the input variable x.\\n        if self.hiddens_hook is not None:\\n            self.hiddens = self.hiddens_hook[1]\\n        else:\\n            self.hiddens = hidden_activation(T.dot(x, W) + b1)\\n        # make the reconstruction (generated) from the hiddens\\n        self.recon_predict = visible_activation(T.dot(self.hiddens, W.T) + b0)\\n        # now compile the predict function accordingly - if it used x or hiddens as the input.\\n        if self.hiddens_hook is not None:\\n            self.f_predict = function(inputs=[self.hiddens], outputs=self.recon_predict)\\n        else:\\n            self.f_predict = function(inputs=[x], outputs=self.recon_predict)\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\nAs you can see, this requires putting in some checks for each hook possibility. However, the modularity this approach offers is worth it, as you will see in the next tutorial. With hooks, you can connect models together in novel ways just like Lego pieces - setting the inputs or hiddens of one model as the outputs of another!\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/DFMktHvlRAyMf744AwzB_lego-297773_640.png\",\n        \"lego-297773_640.png\",\n        \"605\",\n        \"640\",\n        \"#fb8a05\",\n        \"\"\n      ],\n      \"caption\": \"Models connected via hooks.\"\n    }\n  ]\n}\n[/block]\nFinally, to make modularity between models easier, you should define one more method to access the hidden units:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"def get_hiddens(self):\\n    return self.hiddens\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\nThat's it! Now, to put everything together our flexible, modular model looks like:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"import theano.tensor as T\\nfrom opendeep import function\\nfrom opendeep.models.model import Model\\nfrom opendeep.utils.nnet import get_weights_uniform, get_bias\\nfrom opendeep.utils.noise import salt_and_pepper\\nfrom opendeep.utils.activation import get_activation_function\\nfrom opendeep.utils.cost import get_cost_function\\n\\n# create our class initialization!\\nclass DenoisingAutoencoder(Model):\\n    \\\"\\\"\\\"\\n    A denoising autoencoder will corrupt an input (add noise) and try to reconstruct it.\\n    \\\"\\\"\\\"\\n    # Best practice to define all the default parameters up here\\n    # Provide comments when giving parameters so people know what they are for!\\n    _defaults = {\\n        \\\"input_size\\\": 28*28,  # dimensionality of input - works for MNIST\\n        \\\"hidden_size\\\": 1000,  # number of hidden units\\n        \\\"corruption_level\\\": 0.4,  # how much noise to add to the input\\n        \\\"hidden_activation\\\": 'tanh',  # the nonlinearity to apply to hidden units\\n        \\\"visible_activation\\\": 'sigmoid',  # the nonlinearity to apply to visible units\\n        \\\"cost_function\\\": 'binary_crossentropy'  # the cost function to use during training\\n    }\\n    def __init__(self, config=None, defaults=_defaults,\\n                 inputs_hook=None, hiddens_hook=None, params_hook=None,\\n                 input_size=None, hidden_size=None, corruption_level=None,\\n                 hidden_activation=None, visible_activation=None, cost_function=None):\\n        # Now, initialize with Model class to combine config and defaults!\\n        # Here, defaults is defined via a dictionary. However, you could also\\n        # pass a filename to a JSON or YAML file with the same format.\\n        super(DenoisingAutoencoder, self).__init__(\\n            **{arg: val for (arg, val) in locals().iteritems() if arg is not 'self'}\\n        )\\n        # Any parameter from the 'config' will overwrite the 'defaults' dictionary, which will be overwritten if the\\n        # parameter is passed directly to __init__.\\n        # These parameters are now accessible from the 'self' variable!\\n\\n        # Define model hyperparameters\\n        # deal with the inputs_hook and hiddens_hook for the size parameters!\\n        # if the hook exists, grab the size from the first element of the tuple.\\n        if self.inputs_hook is not None:\\n            assert len(self.inputs_hook) == 2, \\\"Was expecting inputs_hook to be a tuple.\\\"\\n            self.input_size = inputs_hook[0]\\n\\n        if self.hiddens_hook is not None:\\n            assert len(self.hiddens_hook) == 2, \\\"was expecting hiddens_hook to be a tuple.\\\"\\n            self.hidden_size = hiddens_hook[0]\\n\\n\\n        # use the helper methods to grab appropriate activation functions from names!\\n        hidden_activation  = get_activation_function(self.hidden_activation)\\n        visible_activation = get_activation_function(self.visible_activation)\\n\\n        # do the same for the cost function\\n        cost_function = get_cost_function(self.cost_function)\\n\\n        # Now, define the symbolic input to the model (Theano)\\n        # We use a matrix rather than a vector so that minibatch processing can be done in parallel.\\n        # Make sure to deal with 'inputs_hook' if it exists!\\n        if self.inputs_hook is not None:\\n            # grab the new input variable from the inputs_hook tuple\\n            x = self.inputs_hook[1]\\n        else:\\n            x = T.fmatrix(\\\"X\\\")\\n        self.inputs = [x]\\n\\n        # Build the model's parameters - a weight matrix and two bias vectors\\n        # Make sure to deal with 'params_hook' if it exists!\\n        if self.params_hook:\\n            # check to see if it contains the three necessary variables\\n            assert len(self.params_hook) == 3, \\\"Not correct number of params to DAE, needs 3!\\\"\\n            W, b0, b1 = self.params_hook\\n        else:\\n            W  = get_weights_uniform(shape=(self.input_size, self.hidden_size), name=\\\"W\\\")\\n            b0 = get_bias(shape=self.input_size, name=\\\"b0\\\")\\n            b1 = get_bias(shape=self.hidden_size, name=\\\"b1\\\")\\n        self.params = [W, b0, b1]\\n\\n        # Perform the computation for a denoising autoencoder!\\n        # first, add noise (corrupt) the input\\n        corrupted_input = salt_and_pepper(input=x, corruption_level=self.corruption_level)\\n        # next, compute the hidden layer given the inputs (the encoding function)\\n        # We don't need to worry about hiddens_hook during training, because we can't\\n        # compute a cost without having the input!\\n        # hiddens_hook is more for the predict function and linking methods below.\\n        hiddens = hidden_activation(T.dot(corrupted_input, W) + b1)\\n        # finally, create the reconstruction from the hidden layer (we tie the weights with W.T)\\n        reconstruction = visible_activation(T.dot(hiddens, W.T) + b0)\\n        # the training cost is reconstruction error\\n        self.train_cost = cost_function(output=reconstruction, target=x)\\n\\n        # Compile everything into a Theano function for prediction!\\n        # When using real-world data in predictions, we wouldn't corrupt the input first.\\n        # Therefore, create another version of the hiddens and reconstruction without adding the noise.\\n        # Here is where we would handle hiddens_hook because this is a generative model!\\n        # For the predict function, it would take in the hiddens instead of the input variable x.\\n        if self.hiddens_hook is not None:\\n            self.hiddens = self.hiddens_hook[1]\\n        else:\\n            self.hiddens = hidden_activation(T.dot(x, W) + b1)\\n        # make the reconstruction (generated) from the hiddens\\n        self.recon_predict = visible_activation(T.dot(self.hiddens, W.T) + b0)\\n        # now compile the predict function accordingly - if it used x or hiddens as the input.\\n        if self.hiddens_hook is not None:\\n            self.f_predict = function(inputs=[self.hiddens], outputs=self.recon_predict)\\n        else:\\n            self.f_predict = function(inputs=[x], outputs=self.recon_predict)\\n\\n    def get_inputs(self):\\n        return self.inputs\\n\\n    def get_hiddens(self):\\n        return self.hiddens\\n\\n    def get_outputs(self):\\n        return self.recon_predict\\n\\n    def predict(self, input):\\n        return self.f_predict(input)\\n\\n    def get_params(self):\\n        return self.params\\n\\n    def get_train_cost(self):\\n        return self.train_cost\\n\\n    def save_args(self, args_file=\\\"dae_config.pkl\\\"):\\n        super(DenoisingAutoencoder, self).save_args(args_file)\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\nWhew, you made it! Not too difficult, but you can see how powerful this makes your model. You are now well-equipped to create awesome new research and contribute models!\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/GRu6bn1wTYCHSj7N9v2N_flash2.gif\",\n        \"flash2.gif\",\n        \"300\",\n        \"167\",\n        \"#8d7861\",\n        \"\"\n      ]\n    }\n  ]\n}\n[/block]","excerpt":"Working with configurations and inputs/hiddens/params hooks! Modularity is key to OpenDeep.","slug":"tutorial-making-your-model-more-flexible","type":"basic","title":"Tutorial: Flexibility and  Modularity"}

Tutorial: Flexibility and Modularity

Working with configurations and inputs/hiddens/params hooks! Modularity is key to OpenDeep.

This tutorial builds on the Denoising Autoencoder built from [Tutorial: Your First Model (DAE)](doc:tutorial-your-first-model). Our naïve implementation of the denoising autoencoder works great for personal use on the MNIST dataset, but it isn't very practical to have a model for one dataset. In this tutorial, you will see how to make a model flexible via configurations, and to deal with hooks for linking models together. #Configurations and defaults With OpenDeep, it is good practice to be able to pass model hyperparameters to the class in a few ways: * a JSON file that will be parsed into a dictionary * a YAML file that will be parsed into a dictionary * a dictionary-like object directly (which means it has a `.get()` method) It would also be good to have a "default" configuration with the ability to override via a "config", or via parameters directly passed into the model. Luckily there is some code to make this really easy! The `__init__` method of the `Model` class handles parsing the "default" and "config" dictionaries as well as the explicitly passed parameters into one variable - `self.args`. You can access any configuration parameters from this variable as well as directly from the `self` reference. Here is an example modifying our class's `__init__` from before to handle configurations of hyperparameters: [block:code] { "codes": [ { "code": "import theano.tensor as T\nfrom opendeep.models.model import Model\nfrom opendeep.utils.nnet import get_weights_uniform, get_bias\nfrom opendeep.utils.noise import salt_and_pepper\nfrom opendeep.utils.activation import get_activation_function\nfrom opendeep.utils.cost import get_cost_function\n\n# create our class initialization!\nclass DenoisingAutoencoder(Model):\n \"\"\"\n A denoising autoencoder will corrupt an input (add noise) and try to reconstruct it.\n \"\"\"\n # Best practice to define all the default parameters up here\n # Provide comments when giving parameters so people know what they are for!\n _defaults = {\n \"input_size\": 28*28, # dimensionality of input - works for MNIST\n \"hidden_size\": 1000, # number of hidden units\n \"corruption_level\": 0.4, # how much noise to add to the input\n \"hidden_activation\": 'tanh', # the nonlinearity to apply to hidden units\n \"visible_activation\": 'sigmoid', # the nonlinearity to apply to visible units\n \"cost_function\": 'binary_crossentropy' # the cost function to use during training\n }\n def __init__(self, config=None, defaults=_defaults, input_size=None, hidden_size=None, corruption_level=None, hidden_activation=None, visible_activation=None, cost_function=None):\n # Now, initialize with Model class with all of the initialization variables passed to the denoising autoencoder, except for the self reference.\n # The parameters will be combined by overriding each other in this order:\n # defaults < config < explicitly passed parameters\n super(DenoisingAutoencoder, self).__init__(\n **{arg: val for (arg, val) in locals().iteritems() if arg is not 'self'}\n )\n # These parameters are now accessible from the 'self.args' dictionary, as well as directly accessing from the 'self' reference!\n\n # use the helper methods to grab appropriate activation functions from names!\n hidden_activation = get_activation_function(self.hidden_activation)\n visible_activation = get_activation_function(self.visible_activation)\n\n # do the same for the cost function\n cost_function = get_cost_function(self.cost_function)\n\n # Now, define the symbolic input to the model (Theano)\n # We use a matrix rather than a vector so that minibatch processing can be done in parallel.\n x = T.fmatrix(\"X\")\n self.inputs = [x]\n\n # Build the model's parameters - a weight matrix and two bias vectors\n W = get_weights_uniform(shape=(self.input_size, self.hidden_size), name=\"W\")\n b0 = get_bias(shape=self.input_size, name=\"b0\")\n b1 = get_bias(shape=self.hidden_size, name=\"b1\")\n self.params = [W, b0, b1]\n\n # Perform the computation for a denoising autoencoder!\n # first, add noise (corrupt) the input\n corrupted_input = salt_and_pepper(input=x, corruption_level=self.corruption_level)\n # next, compute the hidden layer given the inputs (the encoding function)\n hiddens = hidden_activation(T.dot(corrupted_input, W) + b1)\n # finally, create the reconstruction from the hidden layer (we tie the weights with W.T)\n reconstruction = visible_activation(T.dot(hiddens, W.T) + b0)\n # the training cost is reconstruction error\n self.train_cost = cost_function(output=reconstruction, target=x)\n\n # Compile everything into a Theano function for prediction!\n # When using real-world data in predictions, we wouldn't corrupt the input first.\n # Therefore, create another version of the hiddens and reconstruction without adding the noise\n hiddens_predict = hidden_activation(T.dot(x, W) + b1)\n recon_predict = visible_activation(T.dot(hiddens_predict, W.T) + b0)", "language": "python" } ] } [/block] Not too bad! You just made the model infinitely more customizable! #Modularity with other models via hooks Finally, a key concept in OpenDeep is maintaining modularity between models. For this purpose, there are three inputs a model should handle when applicable: * **inputs_hook**: this is a tuple of (size, variable) for an input to your model. It should replace the default input you have set up (in our case, the matrix "X"). * **hiddens_hook**: this is also a tuple of (size, variable) for the hidden representation of your model. It only makes sense in the case of generative models (like denoising autoencoders) where you can compute an output given the hiddens. Therefore hiddens_hook is optional most of the time (unless you have a generative model). This hook is up to your discretion for how the model should handle taking a hidden representation as its input. * **params_hook**: a list of variables to replace the default parameters used by your model. This enables models to be 'tied' together by using the same parameters. In our case, we should handle replacing "W", "b0", and "b1". It is easier to explain via example with comments: [block:code] { "codes": [ { "code": "import theano.tensor as T\nfrom opendeep import function\nfrom opendeep.models.model import Model\nfrom opendeep.utils.nnet import get_weights_uniform, get_bias\nfrom opendeep.utils.noise import salt_and_pepper\nfrom opendeep.utils.activation import get_activation_function\nfrom opendeep.utils.cost import get_cost_function\n\n# create our class initialization!\nclass DenoisingAutoencoder(Model):\n \"\"\"\n A denoising autoencoder will corrupt an input (add noise) and try to reconstruct it.\n \"\"\"\n # Best practice to define all the default parameters up here\n # Provide comments when giving parameters so people know what they are for!\n _defaults = {\n \"input_size\": 28*28, # dimensionality of input - works for MNIST\n \"hidden_size\": 1000, # number of hidden units\n \"corruption_level\": 0.4, # how much noise to add to the input\n \"hidden_activation\": 'tanh', # the nonlinearity to apply to hidden units\n \"visible_activation\": 'sigmoid', # the nonlinearity to apply to visible units\n \"cost_function\": 'binary_crossentropy' # the cost function to use during training\n }\n def __init__(self, config=None, defaults=_defaults,\n inputs_hook=None, hiddens_hook=None, params_hook=None,\n input_size=None, hidden_size=None, corruption_level=None,\n hidden_activation=None, visible_activation=None, cost_function=None):\n # Now, initialize with Model class to combine config and defaults!\n # Here, defaults is defined via a dictionary. However, you could also\n # pass a filename to a JSON or YAML file with the same format.\n super(DenoisingAutoencoder, self).__init__(\n **{arg: val for (arg, val) in locals().iteritems() if arg is not 'self'}\n )\n # Any parameter from the 'config' will overwrite the 'defaults' dictionary, which will be overwritten if the\n # parameter is passed directly to __init__.\n # These parameters are now accessible from the 'self' variable!\n\n # Define model hyperparameters\n # deal with the inputs_hook and hiddens_hook for the size parameters!\n # if the hook exists, grab the size from the first element of the tuple.\n if self.inputs_hook is not None:\n assert len(self.inputs_hook) == 2, \"Was expecting inputs_hook to be a tuple.\"\n self.input_size = inputs_hook[0]\n\n if self.hiddens_hook is not None:\n assert len(self.hiddens_hook) == 2, \"was expecting hiddens_hook to be a tuple.\"\n self.hidden_size = hiddens_hook[0]\n\n\n # use the helper methods to grab appropriate activation functions from names!\n hidden_activation = get_activation_function(self.hidden_activation)\n visible_activation = get_activation_function(self.visible_activation)\n\n # do the same for the cost function\n cost_function = get_cost_function(self.cost_function)\n\n # Now, define the symbolic input to the model (Theano)\n # We use a matrix rather than a vector so that minibatch processing can be done in parallel.\n # Make sure to deal with 'inputs_hook' if it exists!\n if self.inputs_hook is not None:\n # grab the new input variable from the inputs_hook tuple\n x = self.inputs_hook[1]\n else:\n x = T.fmatrix(\"X\")\n self.inputs = [x]\n\n # Build the model's parameters - a weight matrix and two bias vectors\n # Make sure to deal with 'params_hook' if it exists!\n if self.params_hook:\n # check to see if it contains the three necessary variables\n assert len(self.params_hook) == 3, \"Not correct number of params to DAE, needs 3!\"\n W, b0, b1 = self.params_hook\n else:\n W = get_weights_uniform(shape=(self.input_size, self.hidden_size), name=\"W\")\n b0 = get_bias(shape=self.input_size, name=\"b0\")\n b1 = get_bias(shape=self.hidden_size, name=\"b1\")\n self.params = [W, b0, b1]\n\n # Perform the computation for a denoising autoencoder!\n # first, add noise (corrupt) the input\n corrupted_input = salt_and_pepper(input=x, corruption_level=self.corruption_level)\n # next, compute the hidden layer given the inputs (the encoding function)\n # We don't need to worry about hiddens_hook during training, because we can't\n # compute a cost without having the input!\n # hiddens_hook is more for the predict function and linking methods below.\n hiddens = hidden_activation(T.dot(corrupted_input, W) + b1)\n # finally, create the reconstruction from the hidden layer (we tie the weights with W.T)\n reconstruction = visible_activation(T.dot(hiddens, W.T) + b0)\n # the training cost is reconstruction error\n self.train_cost = cost_function(output=reconstruction, target=x)\n\n # Compile everything into a Theano function for prediction!\n # When using real-world data in predictions, we wouldn't corrupt the input first.\n # Therefore, create another version of the hiddens and reconstruction without adding the noise.\n # Here is where we would handle hiddens_hook because this is a generative model!\n # For the predict function, it would take in the hiddens instead of the input variable x.\n if self.hiddens_hook is not None:\n self.hiddens = self.hiddens_hook[1]\n else:\n self.hiddens = hidden_activation(T.dot(x, W) + b1)\n # make the reconstruction (generated) from the hiddens\n self.recon_predict = visible_activation(T.dot(self.hiddens, W.T) + b0)\n # now compile the predict function accordingly - if it used x or hiddens as the input.\n if self.hiddens_hook is not None:\n self.f_predict = function(inputs=[self.hiddens], outputs=self.recon_predict)\n else:\n self.f_predict = function(inputs=[x], outputs=self.recon_predict)", "language": "python" } ] } [/block] As you can see, this requires putting in some checks for each hook possibility. However, the modularity this approach offers is worth it, as you will see in the next tutorial. With hooks, you can connect models together in novel ways just like Lego pieces - setting the inputs or hiddens of one model as the outputs of another! [block:image] { "images": [ { "image": [ "https://files.readme.io/DFMktHvlRAyMf744AwzB_lego-297773_640.png", "lego-297773_640.png", "605", "640", "#fb8a05", "" ], "caption": "Models connected via hooks." } ] } [/block] Finally, to make modularity between models easier, you should define one more method to access the hidden units: [block:code] { "codes": [ { "code": "def get_hiddens(self):\n return self.hiddens", "language": "python" } ] } [/block] That's it! Now, to put everything together our flexible, modular model looks like: [block:code] { "codes": [ { "code": "import theano.tensor as T\nfrom opendeep import function\nfrom opendeep.models.model import Model\nfrom opendeep.utils.nnet import get_weights_uniform, get_bias\nfrom opendeep.utils.noise import salt_and_pepper\nfrom opendeep.utils.activation import get_activation_function\nfrom opendeep.utils.cost import get_cost_function\n\n# create our class initialization!\nclass DenoisingAutoencoder(Model):\n \"\"\"\n A denoising autoencoder will corrupt an input (add noise) and try to reconstruct it.\n \"\"\"\n # Best practice to define all the default parameters up here\n # Provide comments when giving parameters so people know what they are for!\n _defaults = {\n \"input_size\": 28*28, # dimensionality of input - works for MNIST\n \"hidden_size\": 1000, # number of hidden units\n \"corruption_level\": 0.4, # how much noise to add to the input\n \"hidden_activation\": 'tanh', # the nonlinearity to apply to hidden units\n \"visible_activation\": 'sigmoid', # the nonlinearity to apply to visible units\n \"cost_function\": 'binary_crossentropy' # the cost function to use during training\n }\n def __init__(self, config=None, defaults=_defaults,\n inputs_hook=None, hiddens_hook=None, params_hook=None,\n input_size=None, hidden_size=None, corruption_level=None,\n hidden_activation=None, visible_activation=None, cost_function=None):\n # Now, initialize with Model class to combine config and defaults!\n # Here, defaults is defined via a dictionary. However, you could also\n # pass a filename to a JSON or YAML file with the same format.\n super(DenoisingAutoencoder, self).__init__(\n **{arg: val for (arg, val) in locals().iteritems() if arg is not 'self'}\n )\n # Any parameter from the 'config' will overwrite the 'defaults' dictionary, which will be overwritten if the\n # parameter is passed directly to __init__.\n # These parameters are now accessible from the 'self' variable!\n\n # Define model hyperparameters\n # deal with the inputs_hook and hiddens_hook for the size parameters!\n # if the hook exists, grab the size from the first element of the tuple.\n if self.inputs_hook is not None:\n assert len(self.inputs_hook) == 2, \"Was expecting inputs_hook to be a tuple.\"\n self.input_size = inputs_hook[0]\n\n if self.hiddens_hook is not None:\n assert len(self.hiddens_hook) == 2, \"was expecting hiddens_hook to be a tuple.\"\n self.hidden_size = hiddens_hook[0]\n\n\n # use the helper methods to grab appropriate activation functions from names!\n hidden_activation = get_activation_function(self.hidden_activation)\n visible_activation = get_activation_function(self.visible_activation)\n\n # do the same for the cost function\n cost_function = get_cost_function(self.cost_function)\n\n # Now, define the symbolic input to the model (Theano)\n # We use a matrix rather than a vector so that minibatch processing can be done in parallel.\n # Make sure to deal with 'inputs_hook' if it exists!\n if self.inputs_hook is not None:\n # grab the new input variable from the inputs_hook tuple\n x = self.inputs_hook[1]\n else:\n x = T.fmatrix(\"X\")\n self.inputs = [x]\n\n # Build the model's parameters - a weight matrix and two bias vectors\n # Make sure to deal with 'params_hook' if it exists!\n if self.params_hook:\n # check to see if it contains the three necessary variables\n assert len(self.params_hook) == 3, \"Not correct number of params to DAE, needs 3!\"\n W, b0, b1 = self.params_hook\n else:\n W = get_weights_uniform(shape=(self.input_size, self.hidden_size), name=\"W\")\n b0 = get_bias(shape=self.input_size, name=\"b0\")\n b1 = get_bias(shape=self.hidden_size, name=\"b1\")\n self.params = [W, b0, b1]\n\n # Perform the computation for a denoising autoencoder!\n # first, add noise (corrupt) the input\n corrupted_input = salt_and_pepper(input=x, corruption_level=self.corruption_level)\n # next, compute the hidden layer given the inputs (the encoding function)\n # We don't need to worry about hiddens_hook during training, because we can't\n # compute a cost without having the input!\n # hiddens_hook is more for the predict function and linking methods below.\n hiddens = hidden_activation(T.dot(corrupted_input, W) + b1)\n # finally, create the reconstruction from the hidden layer (we tie the weights with W.T)\n reconstruction = visible_activation(T.dot(hiddens, W.T) + b0)\n # the training cost is reconstruction error\n self.train_cost = cost_function(output=reconstruction, target=x)\n\n # Compile everything into a Theano function for prediction!\n # When using real-world data in predictions, we wouldn't corrupt the input first.\n # Therefore, create another version of the hiddens and reconstruction without adding the noise.\n # Here is where we would handle hiddens_hook because this is a generative model!\n # For the predict function, it would take in the hiddens instead of the input variable x.\n if self.hiddens_hook is not None:\n self.hiddens = self.hiddens_hook[1]\n else:\n self.hiddens = hidden_activation(T.dot(x, W) + b1)\n # make the reconstruction (generated) from the hiddens\n self.recon_predict = visible_activation(T.dot(self.hiddens, W.T) + b0)\n # now compile the predict function accordingly - if it used x or hiddens as the input.\n if self.hiddens_hook is not None:\n self.f_predict = function(inputs=[self.hiddens], outputs=self.recon_predict)\n else:\n self.f_predict = function(inputs=[x], outputs=self.recon_predict)\n\n def get_inputs(self):\n return self.inputs\n\n def get_hiddens(self):\n return self.hiddens\n\n def get_outputs(self):\n return self.recon_predict\n\n def predict(self, input):\n return self.f_predict(input)\n\n def get_params(self):\n return self.params\n\n def get_train_cost(self):\n return self.train_cost\n\n def save_args(self, args_file=\"dae_config.pkl\"):\n super(DenoisingAutoencoder, self).save_args(args_file)", "language": "python" } ] } [/block] Whew, you made it! Not too difficult, but you can see how powerful this makes your model. You are now well-equipped to create awesome new research and contribute models! [block:image] { "images": [ { "image": [ "https://files.readme.io/GRu6bn1wTYCHSj7N9v2N_flash2.gif", "flash2.gif", "300", "167", "#8d7861", "" ] } ] } [/block]