1. Visualizing Relationships Between Python Objects Dinu C.
[email protected] Conference 2008-07-08, Vilnius 2. Motivation • Make Python data structures visible • Provide a simple tool for text/book authors • Explore GraphViz capabilities 3. Typical Questions • How does that data structure look like? • How does it change after doing … to it? • What does this module contain? 4. Background 5. Example 1 http://inst.eecs.berkeley.edu/ ~selfpace/cs9honline/Q2/mutation.html 6. Example 2 http://www.linux-user.de/ausgabe/ 2006/09/080-python-1/ 7. Example 3X = [1, 2, 3]L = ['a', X, 'b']D = {'x':X, 'y':2} http://oreilly.com/catalog/9780596513986 8. Example 4 L = ['abc', [(1, 2), ([3], 4)], 5] http://oreilly.com/catalog/9780596513986 9. Example 4 revisitedLWANTED: ref list:01 2 str: abc list:0 1 int: 5 tuple: 0 1 tuple:0 1int: 1int: 2 list: 0int: 4 int: 3 10. Example 3 revisitedLD ref ref list: 0 1 2 dict: keys X valsWANTED: refstr: a str: b list:0 1 2str: y str: xint: 1 int: 3int: 2 11. PDF References (internal) 2:0 1:0 202 19 9:0 8:0 205 193:0106 /Page 5:0 97 7:010:0108 53 11:0 /ExtGState /Page98 24:0 4:0508022:0 20:0 21:0 /Catalog /Pages211 31162114:0 15:06:0 /FontDescriptor109 98 173 /Page/Font25:0/TrueType trailer23:0163 19:03618:09810926:0 /Page 90 13:0 12:0 208 2017:0 16:0 208 20 12. ”First System“os sys errnoposixpath UserDictcopy_regstatImports of module 'os' (simplified) 13. ”First System“fileinfo dirmodmodmod pluginsexpatbuilder guicocoacompatibility guiwxmod mod fileinfo_plugin_xmlmod mod dirminidom modfileinfo_plugin_pyfileinfo_plugin_qtfileinfo_plugin_mp3 xmlmodmodmod mod mod __all__ __all__ mod compiler keyword cStringIO tokenizesax test dommodparsersdir modmod modmodmodpycodegenconsts transformermod expatreader _exceptionspulldom examples fileinfo_plugin_test test_fileinfo minicompatNodeFilter domregxmlbuilder visitormodmodmodfuture symbolssyntax pyassemsymbol parser token fileinfo_plugin_pdf fileinfo_plugin_ttf saxutilsexpat imports docstrings pythoncode unittest fileinfo copy ast new dismisc pyPdf operatorxmlreaderurllib urlparse tracebackplugging investigatorgetopt types pdfgenericsocket string handler linecache globtimedatetime structutilsfilters_ssl_socketosfnmatchre zlib copy_regposixpathUserDictstatModule Space 14. GraphViz 15. Features • Diagrams of abstract graphs and networks • Automatic graph drawing • GUIs, tools, libraries, and language bindings • Developped by AT&T, Open Source • DOT format (text) ➞ ps, pdf, svg, svgz, fig, mif hpgl, pcl, png, gif, dia, imap, cmapx, … 16. Example: Hello World Hello digraph G { Hello -> World } World 17. Example: Hello World 2 Hello digraph G { overlap=false;Hello [shape="box", color="red"]; World [shape="circle", color="blue"];Hello -> World [color="green"]; } World 18. Example: FSM digraph finite_state_machine {rankdir=LR;size="8,5"node [shape = doublecircle]; LR_0 LR_3 LR_4 LR_8;node [shape = circle];LR_0 -> LR_2 [ label = "SS(B)" ];LR_0 -> LR_1 [ label = "SS(S)" ];LR_1 -> LR_3 [ label = "S($end)" ];LR_2 -> LR_6 [ label = "SS(b)" ];LR_2 -> LR_5 [ label = "SS(a)" ];LR_2 -> LR_4 [ label = "S(A)" ];LR_5 -> LR_7 [ label = "S(b)" ];LR_5 -> LR_5 [ label = "S(a)" ];LR_6 -> LR_6 [ label = "S(b)" ];LR_6 -> LR_5 [ label = "S(a)" ];LR_7 -> LR_8 [ label = "S(b)" ];LR_7 -> LR_5 [ label = "S(a)" ];LR_8 -> LR_6 [ label = "S(b)" ];LR_8 -> LR_5 [ label = "S(a)" ]; } 19. Example: FSM LR_4S(A) S(b) S(b) SS(b) LR_6S(a)LR_2 S(a) LR_8SS(B) S(a) SS(a) S(b) LR_0 LR_5 S(b)SS(S) S(a) LR_7 LR_1 S($end) LR_3 20. Pyrels 21. Features • “Second system” – really simple, so far • Recursive func. collecting nodes and edges • Converting to GraphViz nodes and edges 22. Pyrels Profile $ cd pyrels $ fileinfo -p rel -a lc:nmtlines:nclasses:ndefs:ncallsscripts/pyrels $(find src -name "*.py")lc nmtlines nclasses ndefs ncalls path 58 140 19 scripts/pyrels000 00 src/pyrels/__init__.py411 701 7163 src/pyrels/pyrels2dot.py000 00 src/pyrels/test/__init__.py379150010105 src/pyrels/test/pdfposter.py 46 130 2 10 src/pyrels/test/test_all.py 99 682 3 17 src/pyrels/test/test_basic.py 90 261 2 15 src/pyrels/test/test_commandline.py 55 211 2 18 src/pyrels/test/test_cyles.py148 621 6 63 src/pyrels/test/test_multi.py193 66514 50 src/pyrels/test/test_other.py 2070 26 src/pyrels/test/testutils.py 1240 11 src/pyrels/utils.py 1511501 1150457 total 23. Example: String Cachingref 1v str: XXXXXXXXXXXXXXXXXXXXX v = 'X' * 21 refu str: XXXXXXXXXXXXXXXXXXXXX u = 'X' * 21 s = 'X' * 20 3 t = 'X' * 20 s ref str: XXXXXXXXXXXXXXXXXXXXref t 2 digraph G { name1775744 [label="v", shape="ellipse"]; str3376528 [label="str: XXXXXXXXXXXXXXXXXXXXX", shape="box"]; name1845536 [label="u", shape="ellipse"]; str3284144 [label="str: XXXXXXXXXXXXXXXXXXXXX", shape="box"]; name1686816 [label="s", shape="ellipse"]; str3376480 [label="str: XXXXXXXXXXXXXXXXXXXX", shape="box"]; name1798720 [label="t", shape="ellipse"]; str3376480 [label="str: XXXXXXXXXXXXXXXXXXXX", shape="box"];name1686816 -> str3376480 [label="ref"]; name1845536 -> str3284144 [label="ref"]; name1798720 -> str3376480 [label="ref"]; name1775744 -> str3376528 [label="ref"]; } 24. Example: Singleton Nonedigraph G { 1 2name3498784 [label="aDict", shape="ellipse"];dict3660224 [label="dict: | {keys|vals}|{|}", shape="record"]; none1 = None none [label="None", shape="box"]; none2 = None none [label="None", shape="box"]; aList = [None] name3499008 [label="aList", shape="ellipse"];list3662272 [label="list: | 0", aTuple = (None, None)shape="record"]; aDict = {None:None}none [label="None", shape="box"];name3498976 [label="none2", shape="ellipse"];none [label="None", shape="box"];dict: name3498944 [label="none1", shape="ellipse"];aDict ref keys valsnone [label="None", shape="box"];name3317728 [label="aTuple",shape="ellipse"];tuple3334392 [label="tuple: | 0| 1",list: shape="Mrecord"]; refaList none [label="None", shape="box"];0 none [label="None", shape="box"];none2 ref None dict3660224:k0 -> none [label=""]; refdict3660224:v0 -> none [label=""];3 list3662272:0 -> none [label=""]; none1tuple3334392:0 -> none [label=""];tuple3334392:1 -> none [label=""];name3498784 -> dict3660224 [label="ref"]; tuple: name3499008 -> list3662272 [label="ref"];aTuple ref0 name3498976 -> none [label="ref"];name3498944 -> none [label="ref"];1 name3317728 -> tuple3334392 [label="ref"];} 25. Example: Module Content (pyPdf) ref utilsmodule: pyPdf.utils import pyPdf ref locals().update(pyPdf.__dict__) __doc__None del pyPdf ref pdf module: pyPdf.pdf ref PdfFileWriter new style class: PdfFileWriter ref__name__ str: 'pyPdf' reffiltersmodule: pyPdf.filters ref __file__ str: '/usr/local/lib/python2.5/site-packages/pyPdf/__init__.pyc' refgenericmodule: pyPdf.generic ref list: __path__0 str: '/usr/local/lib/python2.5/site-packages/pyPdf' ref list:__all__0str: 'pdf'ref PdfFileReader new style class: PdfFileReader 26. Future 27. Issues • Testing, debugging & refactoring, … • Detecting cycles • Fiddling with GraphViz params (overlap, …) • Adding other relationships:- Module Imports or not)- Inheritance (UML-like- Call graphs-… • Smart graph pruning 28. Summary • Development only barely started (v. 0.1) • Useful already for visualising data structures • Provides certain insights for Python newbies • Feedback is warmly welcome! 29. Links • http://www.dinu-gherman.net/tmp/pyrels-0.1.0.tar.gz• http://www.graphviz.org 30. Questions? 31. Questions! • GraphViz editor with branch fold-in/out? • How to get full path for DOT edges (in plain format)?