{"_id":"563fc7641594380d009c1a60","__v":6,"project":"5503ea178c5e913700362c70","version":{"_id":"563fc7631594380d009c1a5c","project":"5503ea178c5e913700362c70","__v":2,"createdAt":"2015-11-08T22:06:27.279Z","releaseDate":"2015-11-08T22:06:27.278Z","categories":["563fc7641594380d009c1a5d","563fc7641594380d009c1a5e","563fc7641594380d009c1a5f","5654ff257b89070d00f96386"],"is_deprecated":false,"is_hidden":false,"is_beta":true,"is_stable":true,"codename":"","version_clean":"0.0.9","version":"0.0.9"},"category":{"_id":"563fc7641594380d009c1a5f","__v":1,"pages":["563fc7641594380d009c1a60"],"project":"5503ea178c5e913700362c70","version":"563fc7631594380d009c1a5c","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2015-04-17T17:03:23.681Z","from_sync":false,"order":3,"slug":"guides","title":"Guides"},"user":"5503e897e508a017002013bd","updates":["5715bbc0b5af590e00a0fd87"],"next":{"pages":[],"description":""},"createdAt":"2015-04-17T17:05:13.667Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"settings":"","results":{"codes":[]},"auth":"required","params":[],"url":""},"isReference":false,"order":0,"body":"Keeping track of your model's performance is essential for debugging and evaluation. OpenDeep provides a flexible way to monitor variables and expressions that you may care about with three classes:\n\n* Monitor (opendeep.monitor.monitor)\n* MonitorsChannel (opendeep.monitor.monitor)\n* Plot (opendeep.monitor.plot)\n\nIn this guide, we will go over the usage of these three classes and show an example plotting train cost, classification error, and mean weights for a simple MLP.\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Monitor (opendeep.monitor.monitor)\"\n}\n[/block]\nThe Monitor class is the basis for indicating what variables and expressions you want to see. \n\nIt is defined by a name, expression to evaluate, and three flags (train, valid, test) to indicate which subsets of your data to use with the monitor. By default, the train flag is true while valid and test are false.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"from opendeep.monitor.monitor import Monitor\\n\\nyour_monitor = Monitor(\\n  name=\\\"informative_name\\\",\\n  expression=theano_expression_or_variable,\\n  train=True,\\n  valid=False,\\n  test=True\\n)\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"MonitorsChannel (opendeep.monitor.monitor)\"\n}\n[/block]\nThe MonitorsChannel is a logical grouping of Monitor objects. It acts as an organizational helper. You can think of these channels as figures or plots, where the individual monitors in the channel are plotted in the same figure.\n\nIt is defined by a name and list of monitors, and has helper functions for accessing or modifying its monitors:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"from opendeep.monitor.monitor import MonitorsChannel\\n\\n# you can instantiate with a list of monitors, or add them later.\\n# instantiate with list:\\nyour_channel = MonitorsChannel(\\n  name=\\\"your_channel_name\\\",\\n  monitors=[monitor_1, monitor_2]\\n)\\n\\n# adding monitors\\nyour_channel.add(monitor_3)\\nyour_channel.add([monitor_4, monitor_5])\\n\\n# accessing (and removing) a monitor with a specific name from the channel:\\nmonitor_4 = your_channel.pop(\\\"monitor_4_name\\\")\\n# or just removing\\nyour_channel.remove(\\\"monitor_3_name\\\")\\n\\n# getting the list of monitor names\\nmonitor_names = your_channel.get_monitor_names()\\n# getting the list of monitor expressions\\nmonitor_expressions = your_channel.get_monitor_expressions()\\n\\n# getting the Monitor objects that should be evaluated on the train subset\\ntrain_monitors = your_channel.get_train_monitors()\\n# for valid subset\\nvalid_monitors = your_channel.get_valid_monitors()\\n# for test\\ntest_monitors  = your_channel.get_test_monitors()\\n\\n# accessing Monitors from multiple data subsets at once (where Monitors are not duplicated)\\ntrain_and_test = your_channel.get_monitors(train=True, test=True)\\nvalid_and_test = your_channel.get_monitors(valid=True, test=True)\\n# or just return them all!\\nall_monitors = your_channel.get_monitors()\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Plot (opendeep.monitor.plot)\"\n}\n[/block]\nThe Plot object interfaces with [Bokeh-server](http://bokeh.pydata.org/en/latest/docs/user_guide/server.html). It uses a list of MonitorsChannel or Monitor objects to construct a separate figure for each channel, where the monitors within the channel get plotted on the figure. To use the plot, pass it to your Optimizer's train() function! If provided with just a Monitor, the plot treats it as a channel with the given monitor as the only object.\n\nThe Plot object will always have a train_cost figure to keep track of your training cost expression, so you never have to create a Monitor for it. The Plot is defined by:\n* bokeh_doc_name: name for the Bokeh document\n* monitor_channels: the list of MonitorsChannel or Monitor to plot\n* open_browser (optional): whether to open a browser to the figures immediately\n* start_server (optional): whether to start the bokeh server for you (not recommended, you should instead start it from the command line yourself with the command `bokeh-server`)\n* server_url (optional): the URL to use when connecting to the bokeh server\n* colors (optional): a list of different possible color hex codes to use for plotting (so multiple lines are legible on the same plot)\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"from opendeep.monitor.plot import Plot\\n\\nyour_plot = Plot(\\n  bokeh_doc_name=\\\"Your_Awesome_Plot\\\",\\n  monitor_channels=[your_channel_1, your_channel_2],\\n  open_browser=True,\\n  start_server=False,\\n  server_url='http://localhost:5006/',\\n  colors=['#1f77b4', '#ff7f0e', '#2ca02c', '#d62728']\\n)\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Example: Plotting train cost, softmax classification error, and mean weights for a simple MLP\"\n}\n[/block]\nThis tutorial isn't focused on the model, so let's do something really simple. If you want an overview of the MLP, check out [Classifying Handwritten MNIST Images](doc:tutorial-classifying-handwritten-mnist-images).\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"from opendeep.models import Prototype, Dense, Softmax\\nfrom opendeep.optimization.loss import Neg_LL\\nfrom opendeep.optimization import AdaDelta\\nfrom opendeep.data import MNIST\\nfrom theano.tensor import matrix, lvector\\n\\n# create a 1-layer MLP\\nmlp = Prototype()\\nmlp.add(Dense(inputs=((None, 28*28), matrix('xs')), outputs=512)\\nmlp.add(Softmax, outputs=10, out_as_probs=False)\\n\\n# define our loss\\nloss = Neg_LL(inputs=mlp.models[-1].p_y_given_x, targets=lvector('ys'), one_hot=False)\\n\\n# make our optimizer object with the mlp on the MNIST dataset\\noptimizer = AdaDelta(model=mlp, loss=loll, dataset=MNIST(), epochs=20)\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\nNow our setup is out of the way, let's create some monitors and monitor channels to plot during out training!\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"import theano.tensor as T\\nfrom opendeep.monitor.monitor import Monitor, MonitorsChannel\\n\\n# Now, let's create monitors to look at some weights statistics\\nW1 = mlp[0].get_params()['W']\\nW2 = mlp[1].get_params()['W']\\n\\nmean_W1 = T.mean(W1)\\nmean_W2 = T.mean(W2)\\n\\nmean_W1_monitor = Monitor(name=\\\"hiddens_mean\\\", expression=mean_W1)\\nmean_W2_monitor = Monitor(name=\\\"out_mean\\\", expression=mean_W2)\\n\\n# create a new channel from the relevant weights monitors\\nweights_channel = MonitorsChannel(name=\\\"weights\\\", monitors=[mean_W1_monitor, mean_W2_monitor])\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\nFinally we can create a Plot object to visualize these monitors:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"from opendeep.monitor.plot import Plot\\n\\n# create our plot!\\nplot = Plot(bokeh_doc_name=\\\"Monitor_Tutorial\\\", monitor_channels=[mlp_channel, weights_channel], open_browser=True)\\n\\n# train our model and use the plot (make sure you start bokeh-server first!):\\noptimizer.train(plot=plot)\\n\\n# that's it!\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\nPutting it all together:\n\n1) Start Bokeh server\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"bokeh serve\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\n2) Run your python script with the model\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"import theano.tensor as T\\nfrom opendeep.models import Prototype, Dense, Softmax\\nfrom opendeep.optimization.loss import Neg_LL\\nfrom opendeep.optimization import AdaDelta\\nfrom opendeep.data import MNIST\\nfrom opendeep.monitor.plot import Plot\\nfrom opendeep.monitor.monitor import Monitor, MonitorsChannel\\n\\ndef main():\\n  # create a 2-layer MLP\\n  mlp = Prototype()\\n  mlp.add(Dense(inputs=((None, 28*28), T.matrix('xs')), outputs=512))\\n  mlp.add(Softmax, outputs=10, out_as_probs=True)\\n\\n  # define our loss\\n  loss = Neg_LL(inputs=mlp.get_outputs(), targets=T.lvector('ys'), one_hot=False)\\n\\n  # make our optimizer object with the mlp on the MNIST dataset\\n  optimizer = AdaDelta(model=mlp, loss=loss, dataset=MNIST(), epochs=20)\\n\\n  # create the monitors and channels\\n  W1 = mlp[0].get_param('W')\\n  W2 = mlp[1].get_param('W')\\n\\n  mean_W1 = T.mean(W1)\\n  mean_W2 = T.mean(W2)\\n\\n  mean_W1_monitor = Monitor(name=\\\"hiddens_mean\\\", expression=mean_W1)\\n  mean_W2_monitor = Monitor(name=\\\"out_mean\\\", expression=mean_W2)\\n\\n  weights_channel = MonitorsChannel(name=\\\"weights\\\", monitors=[mean_W1_monitor, mean_W2_monitor])\\n\\n  # create the plot\\n  plot = Plot(bokeh_doc_name=\\\"Monitor_Tutorial\\\", monitor_channels=weights_channel, open_browser=True)\\n\\n  # train our model!\\n  optimizer.train(plot=plot)\\n\\n\\nif __name__ == '__main__':\\n    main()\",\n      \"language\": \"python\"\n    }\n  ]\n}\n[/block]\nCongrats! You should get output graphs similar to the following (these were created with slightly different code):\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/RisxSRHkTnyqOO8ilIKR_monitor_tutorial_train.png\",\n        \"monitor_tutorial_train.png\",\n        \"600\",\n        \"600\",\n        \"#6e8594\",\n        \"\"\n      ],\n      \"caption\": \"MLP train cost.\"\n    }\n  ]\n}\n[/block]\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/qLyk6TH7S52FQJGKp3mh_monitor_tutorial_error.png\",\n        \"monitor_tutorial_error.png\",\n        \"600\",\n        \"600\",\n        \"#2f4a5b\",\n        \"\"\n      ],\n      \"caption\": \"MLP softmax classification error.\"\n    }\n  ]\n}\n[/block]\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/ZEtNLpASVi37K6otzlSw_monitor_tutorial_weights.png\",\n        \"monitor_tutorial_weights.png\",\n        \"600\",\n        \"600\",\n        \"#3e88be\",\n        \"\"\n      ],\n      \"caption\": \"MLP mean weights.\"\n    }\n  ]\n}\n[/block]","excerpt":"Keeping track of values during training/testing.","slug":"monitors-and-live-plotting","type":"basic","title":"Monitors and Live Plotting"}

Monitors and Live Plotting

Keeping track of values during training/testing.

Keeping track of your model's performance is essential for debugging and evaluation. OpenDeep provides a flexible way to monitor variables and expressions that you may care about with three classes: * Monitor (opendeep.monitor.monitor) * MonitorsChannel (opendeep.monitor.monitor) * Plot (opendeep.monitor.plot) In this guide, we will go over the usage of these three classes and show an example plotting train cost, classification error, and mean weights for a simple MLP. [block:api-header] { "type": "basic", "title": "Monitor (opendeep.monitor.monitor)" } [/block] The Monitor class is the basis for indicating what variables and expressions you want to see. It is defined by a name, expression to evaluate, and three flags (train, valid, test) to indicate which subsets of your data to use with the monitor. By default, the train flag is true while valid and test are false. [block:code] { "codes": [ { "code": "from opendeep.monitor.monitor import Monitor\n\nyour_monitor = Monitor(\n name=\"informative_name\",\n expression=theano_expression_or_variable,\n train=True,\n valid=False,\n test=True\n)", "language": "python" } ] } [/block] [block:api-header] { "type": "basic", "title": "MonitorsChannel (opendeep.monitor.monitor)" } [/block] The MonitorsChannel is a logical grouping of Monitor objects. It acts as an organizational helper. You can think of these channels as figures or plots, where the individual monitors in the channel are plotted in the same figure. It is defined by a name and list of monitors, and has helper functions for accessing or modifying its monitors: [block:code] { "codes": [ { "code": "from opendeep.monitor.monitor import MonitorsChannel\n\n# you can instantiate with a list of monitors, or add them later.\n# instantiate with list:\nyour_channel = MonitorsChannel(\n name=\"your_channel_name\",\n monitors=[monitor_1, monitor_2]\n)\n\n# adding monitors\nyour_channel.add(monitor_3)\nyour_channel.add([monitor_4, monitor_5])\n\n# accessing (and removing) a monitor with a specific name from the channel:\nmonitor_4 = your_channel.pop(\"monitor_4_name\")\n# or just removing\nyour_channel.remove(\"monitor_3_name\")\n\n# getting the list of monitor names\nmonitor_names = your_channel.get_monitor_names()\n# getting the list of monitor expressions\nmonitor_expressions = your_channel.get_monitor_expressions()\n\n# getting the Monitor objects that should be evaluated on the train subset\ntrain_monitors = your_channel.get_train_monitors()\n# for valid subset\nvalid_monitors = your_channel.get_valid_monitors()\n# for test\ntest_monitors = your_channel.get_test_monitors()\n\n# accessing Monitors from multiple data subsets at once (where Monitors are not duplicated)\ntrain_and_test = your_channel.get_monitors(train=True, test=True)\nvalid_and_test = your_channel.get_monitors(valid=True, test=True)\n# or just return them all!\nall_monitors = your_channel.get_monitors()", "language": "python" } ] } [/block] [block:api-header] { "type": "basic", "title": "Plot (opendeep.monitor.plot)" } [/block] The Plot object interfaces with [Bokeh-server](http://bokeh.pydata.org/en/latest/docs/user_guide/server.html). It uses a list of MonitorsChannel or Monitor objects to construct a separate figure for each channel, where the monitors within the channel get plotted on the figure. To use the plot, pass it to your Optimizer's train() function! If provided with just a Monitor, the plot treats it as a channel with the given monitor as the only object. The Plot object will always have a train_cost figure to keep track of your training cost expression, so you never have to create a Monitor for it. The Plot is defined by: * bokeh_doc_name: name for the Bokeh document * monitor_channels: the list of MonitorsChannel or Monitor to plot * open_browser (optional): whether to open a browser to the figures immediately * start_server (optional): whether to start the bokeh server for you (not recommended, you should instead start it from the command line yourself with the command `bokeh-server`) * server_url (optional): the URL to use when connecting to the bokeh server * colors (optional): a list of different possible color hex codes to use for plotting (so multiple lines are legible on the same plot) [block:code] { "codes": [ { "code": "from opendeep.monitor.plot import Plot\n\nyour_plot = Plot(\n bokeh_doc_name=\"Your_Awesome_Plot\",\n monitor_channels=[your_channel_1, your_channel_2],\n open_browser=True,\n start_server=False,\n server_url='http://localhost:5006/',\n colors=['#1f77b4', '#ff7f0e', '#2ca02c', '#d62728']\n)", "language": "python" } ] } [/block] [block:api-header] { "type": "basic", "title": "Example: Plotting train cost, softmax classification error, and mean weights for a simple MLP" } [/block] This tutorial isn't focused on the model, so let's do something really simple. If you want an overview of the MLP, check out [Classifying Handwritten MNIST Images](doc:tutorial-classifying-handwritten-mnist-images). [block:code] { "codes": [ { "code": "from opendeep.models import Prototype, Dense, Softmax\nfrom opendeep.optimization.loss import Neg_LL\nfrom opendeep.optimization import AdaDelta\nfrom opendeep.data import MNIST\nfrom theano.tensor import matrix, lvector\n\n# create a 1-layer MLP\nmlp = Prototype()\nmlp.add(Dense(inputs=((None, 28*28), matrix('xs')), outputs=512)\nmlp.add(Softmax, outputs=10, out_as_probs=False)\n\n# define our loss\nloss = Neg_LL(inputs=mlp.models[-1].p_y_given_x, targets=lvector('ys'), one_hot=False)\n\n# make our optimizer object with the mlp on the MNIST dataset\noptimizer = AdaDelta(model=mlp, loss=loll, dataset=MNIST(), epochs=20)", "language": "python" } ] } [/block] Now our setup is out of the way, let's create some monitors and monitor channels to plot during out training! [block:code] { "codes": [ { "code": "import theano.tensor as T\nfrom opendeep.monitor.monitor import Monitor, MonitorsChannel\n\n# Now, let's create monitors to look at some weights statistics\nW1 = mlp[0].get_params()['W']\nW2 = mlp[1].get_params()['W']\n\nmean_W1 = T.mean(W1)\nmean_W2 = T.mean(W2)\n\nmean_W1_monitor = Monitor(name=\"hiddens_mean\", expression=mean_W1)\nmean_W2_monitor = Monitor(name=\"out_mean\", expression=mean_W2)\n\n# create a new channel from the relevant weights monitors\nweights_channel = MonitorsChannel(name=\"weights\", monitors=[mean_W1_monitor, mean_W2_monitor])", "language": "python" } ] } [/block] Finally we can create a Plot object to visualize these monitors: [block:code] { "codes": [ { "code": "from opendeep.monitor.plot import Plot\n\n# create our plot!\nplot = Plot(bokeh_doc_name=\"Monitor_Tutorial\", monitor_channels=[mlp_channel, weights_channel], open_browser=True)\n\n# train our model and use the plot (make sure you start bokeh-server first!):\noptimizer.train(plot=plot)\n\n# that's it!", "language": "python" } ] } [/block] Putting it all together: 1) Start Bokeh server [block:code] { "codes": [ { "code": "bokeh serve", "language": "shell" } ] } [/block] 2) Run your python script with the model [block:code] { "codes": [ { "code": "import theano.tensor as T\nfrom opendeep.models import Prototype, Dense, Softmax\nfrom opendeep.optimization.loss import Neg_LL\nfrom opendeep.optimization import AdaDelta\nfrom opendeep.data import MNIST\nfrom opendeep.monitor.plot import Plot\nfrom opendeep.monitor.monitor import Monitor, MonitorsChannel\n\ndef main():\n # create a 2-layer MLP\n mlp = Prototype()\n mlp.add(Dense(inputs=((None, 28*28), T.matrix('xs')), outputs=512))\n mlp.add(Softmax, outputs=10, out_as_probs=True)\n\n # define our loss\n loss = Neg_LL(inputs=mlp.get_outputs(), targets=T.lvector('ys'), one_hot=False)\n\n # make our optimizer object with the mlp on the MNIST dataset\n optimizer = AdaDelta(model=mlp, loss=loss, dataset=MNIST(), epochs=20)\n\n # create the monitors and channels\n W1 = mlp[0].get_param('W')\n W2 = mlp[1].get_param('W')\n\n mean_W1 = T.mean(W1)\n mean_W2 = T.mean(W2)\n\n mean_W1_monitor = Monitor(name=\"hiddens_mean\", expression=mean_W1)\n mean_W2_monitor = Monitor(name=\"out_mean\", expression=mean_W2)\n\n weights_channel = MonitorsChannel(name=\"weights\", monitors=[mean_W1_monitor, mean_W2_monitor])\n\n # create the plot\n plot = Plot(bokeh_doc_name=\"Monitor_Tutorial\", monitor_channels=weights_channel, open_browser=True)\n\n # train our model!\n optimizer.train(plot=plot)\n\n\nif __name__ == '__main__':\n main()", "language": "python" } ] } [/block] Congrats! You should get output graphs similar to the following (these were created with slightly different code): [block:image] { "images": [ { "image": [ "https://files.readme.io/RisxSRHkTnyqOO8ilIKR_monitor_tutorial_train.png", "monitor_tutorial_train.png", "600", "600", "#6e8594", "" ], "caption": "MLP train cost." } ] } [/block] [block:image] { "images": [ { "image": [ "https://files.readme.io/qLyk6TH7S52FQJGKp3mh_monitor_tutorial_error.png", "monitor_tutorial_error.png", "600", "600", "#2f4a5b", "" ], "caption": "MLP softmax classification error." } ] } [/block] [block:image] { "images": [ { "image": [ "https://files.readme.io/ZEtNLpASVi37K6otzlSw_monitor_tutorial_weights.png", "monitor_tutorial_weights.png", "600", "600", "#3e88be", "" ], "caption": "MLP mean weights." } ] } [/block]