easy_install compatibility

setuptools_optional.patch.txt

This would fix #857, if setuptools were importable at build time.

Just in case it wasn't clear, this patch does not cause Nevow to require setuptools, nor change the behavior of ./setup.py when setuptools is absent. If there's anything else you want, let me know.

I'm very interested in getting this patch into mainline Nevow so that I can rely on it when re-using Nevow in allmydata.org, so please let me know if there is anything about this patch that you would like to be different.

It's not obvious how to automatically test "what happens when you install nevow". It seems slightly weird to test that in nevow's unit tests -- for example the test would have to be careful not to let the nevow that is imported be used when it is trying to test the nevow that gets installed. It would probably want to do everything in a subprocess, i.e. something like:

tar xzvf dists/nevow-$VER.tar.gz
cd dists/nevow-$VER
python setup.py install --prefix=./testinst
PYTHONPATH=./testinst/lib/python2.5/site-packages python -c 'import nevow'

Now we already have a test which does this -- tests installation of nevow -- in http://allmydata.org . Here is the source code for that test. You can see the results on [the allmydata.org buildbot http://allmydata.org/buildbot/waterfall] -- many platforms have a "TODO" result for that test, and some have a "SUCCESS?!?" result.

This is related to twisted #2308 (Test Twisted's release automation).

I have a vague notion to write shell scripts to do the above script in a Nexenta Zone, but I don't understand how such a thing could become a standard part of the upstream twisted/nevow unit tests.

Conversation with dreid on IRC enlightened me about an idea that exarkun had previously suggested. My question was: where resides the source code for the unit tests to test installation? It doesn't really make sense for that code to reside in the unit tests of the package itself.

The answer is: in the master.cfg of the buildmaster. There should be a buildstep which installs the build package into a local directory and runs the unit tests on the code thus installed.

see also allmydata.org "Tahoe" #348 (BuildBot step to run tests from package)

First step here is to get an easy_install buildslave set up (perhaps a regular install buildslave, too).

Twisted itself has "easy_install buildslaves", like this one:

http://buildbot.twistedmatrix.com/builders/debian-easy-py2.5-epoll/builds/107/steps/shell_5/logs/stdio

I would be happy to run such a buildslave (or several). I would also be happy to edit the nevow master.cfg file to have code much like the twisted master.cfg file's code to do these easy_install tests. To do either of these things I need help from someone who has access to the nevow buildmaster.

Could someone please give me a copy of the twisted master.cfg and the nevow master.cfg?

Hey there, I'd really like to get this issue fixed so that certain "TODO" stack traces stop appearing in the Tahoe Least-Authority Filesystem unit tests and runtime logs, e.g.:

http://allmydata.org/buildbot/builders/jrydberg%20osx%2010.5/builds/71/steps/test/logs/stdio

I've tested this patch in the Tahoe unit tests and demonstrated that it fixes this problem. I would be happy to help edit the Nevow buildmaster config to make the Nevow buildbot test this use case automatically. What can I do to help?

I open ticket #2630 (installation of Nevow doesn't have automated tests).

Okay, ticket #2630 is done -- there is automated testing of installing Nevow with easy_install. Now, please apply this patch to simplify setup.py, make Nevow able to find its package data files at runtime even when it is in a .zip file, and make it possible for Nevow to programmatically declare its dependency on Twisted. Thanks!

(In [16177]) Branching to 'nevow-install-2527'

Getting this error upon running the tests with this patch applied:

===============================================================================
[ERROR]: nevow.test.test_athena.Utilities.testPackage

Traceback (most recent call last):
  File "/home/buildbot/q-exponent/q-nevowinstall/build/installdir/lib/python2.4/site-packages/Nevow-0.9.31_r16178-py2.4.egg/nevow/test/test_athena.py", line 191, in testPackage
    m = package.mapping.pop(module)
exceptions.KeyError: 'pop(): dictionary is empty'
-------------------------------------------------------------------------------

(I've created a branch for this patch: /branches/nevow-install-2527.)

(In [16241]) Branching to 'nevow-setuptools-compat-2527'

Is it possible to get the divmod trac to mail me when this ticket changes? (And same for any ticket that I've touched.)

I believe this requires configuring the "smtp" options in the trac.ini.

How do I check out this branch from SVN? The following attempt failed:

svn co http://divmod.org/svn/Divmod/branches/nevow-setuptools-compat-2527

The branch name was wrong. It should be nevow-install-2527.

I can't reproduce your test failure, washort.

When I run ./setup.py clean && ./setup.py build && PYTHONPATH=./build/lib/ trial nevow on this branch then tests pass:

PASSED (skips=17, expectedFailures=2, successes=781)

Note that I removed setuptools from my system in order to run this test, so I'm testing the condition in the setup.py where setuptools cannot be imported. After reinstalling the Ubuntu Hardy package of python-setuptools, I got the same result.

This change seems to improve installation significantly, but it still doesn't produce a working install. It misses the twistd plugin in Nevow/twisted/plugins/. Any idea how to get this installed?

I'm not familiar with the twistd plugin architecture. Is the problem caused by this permissions failure:

trial --reporter=bwverbose --reactor=select nevow formless
 in dir /home/divmod/twisted-trunk/q-nevowinstall/build/installdir (timeout 1200 secs)
 watching logfiles {'test.log': '_trial_temp/test.log'}
 argv: ['trial', '--reporter=bwverbose', '--reactor=select', 'nevow', 'formless']
 environment: {'LANG': 'en_US.UTF-8', 'TERM': 'xterm-color', 'SHELL': '/bin/bash', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'LANGUAGE': 'en_US:en_GB:en', 'SHLVL': '1', 'PYTHONPATH': 'lib/python2.5/site-packages', 'OLDPWD': '/home/divmod', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'PWD': '/home/divmod/twisted-trunk', 'LOGNAME': 'divmod', 'USER': 'divmod', 'MAIL': '/var/mail/divmod', 'PATH': '/usr/local/bin:/usr/bin:/bin:/usr/games', 'LS_COLORS': 'no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.flac=01;35:*.mp3=01;35:*.mpc=01;35:*.ogg=01;35:*.wav=01;35:', 'HOME': '/home/divmod', '_': '/usr/bin/python2.4'}
/usr/lib/python2.5/site-packages/zope/__init__.py:19: UserWarning: Module twisted was already imported from /usr/lib/python2.5/site-packages/twisted/__init__.pyc, but /home/divmod/twisted-trunk/q-nevowinstall/build/installdir/lib/python2.5/site-packages/Nevow-0.9.32_r16888-py2.5.egg is being added to sys.path
  import pkg_resources
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/twisted/python/usage.py", line 373, in <lambda>
    fn = lambda name, value, m=method: m(value)
  File "/usr/lib/python2.5/site-packages/twisted/application/app.py", line 488, in opt_reactor
    installReactor(shortName)
  File "/usr/lib/python2.5/site-packages/twisted/application/reactors.py", line 79, in installReactor
    for installer in getReactorTypes():
  File "/usr/lib/python2.5/site-packages/twisted/plugin.py", line 200, in getPlugins
    allDropins = getCache(package)
--- <exception caught here> ---
  File "/usr/lib/python2.5/site-packages/twisted/plugin.py", line 179, in getCache
    dropinPath.setContent(pickle.dumps(dropinDotCache))
  File "/usr/lib/python2.5/site-packages/twisted/python/filepath.py", line 574, in setContent
    sib.open('w').write(content)
  File "/usr/lib/python2.5/site-packages/twisted/python/filepath.py", line 384, in open
    return open(self.path, mode+'b')
exceptions.IOError: [Errno 13] Permission denied: '/usr/lib/python2.5/site-packages/twisted/plugins/dropin.cache.new'
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/twisted/python/usage.py", line 241, in parseOptions
    self.postOptions()
  File "/usr/lib/python2.5/site-packages/twisted/scripts/trial.py", line 293, in postOptions
    self['reporter'] = self._loadReporterByName(self['reporter'])
  File "/usr/lib/python2.5/site-packages/twisted/scripts/trial.py", line 279, in _loadReporterByName
    for p in plugin.getPlugins(itrial.IReporter):
  File "/usr/lib/python2.5/site-packages/twisted/plugin.py", line 200, in getPlugins
    allDropins = getCache(package)
--- <exception caught here> ---
  File "/usr/lib/python2.5/site-packages/twisted/plugin.py", line 179, in getCache
    dropinPath.setContent(pickle.dumps(dropinDotCache))
  File "/usr/lib/python2.5/site-packages/twisted/python/filepath.py", line 574, in setContent
    sib.open('w').write(content)
  File "/usr/lib/python2.5/site-packages/twisted/python/filepath.py", line 384, in open
    return open(self.path, mode+'b')
exceptions.IOError: [Errno 13] Permission denied: '/usr/lib/python2.5/site-packages/twisted/plugins/dropin.cache.new'

(from http://buildbot.divmod.org/builders/linux32-py2.5-nevowinstall/builds/104/steps/select_2/logs/stdio)

?

It obvious to me from looking at that stack trace if the solution to this permissions failure is "give it permission to do that" or "it shouldn't need to do that".

Note to self: once this ticket is fixed, work on #2629 (Nevow doesn't declare its dependency on Twisted in a machine-parseable way).

Well, I guess the permission denied traceback isn't a problem, since it also occurs in these tests, which pass:

http://buildbot.divmod.org/builders/linux32-py2.5-nevowinstall/builds/104/steps/trial/logs/stdio

Okay, thanks to exarkun for explaining to me what the problem was.

The patch makes it so that if setuptools is present (importable) then it uses the setuptools "find_packages()" feature instead of having a list of packages in the setup.py. setuptools doesn't perceive twisted/plugins as being a package (presumably because it doesn't have an init.py in it), so it is excluded from the things to package, and thus the resulting error.

Here is a patch that fixes this by explicitly listing "twisted.plugins" as a package to be added to the list of packages discovered by setuptools. Alternate solutions would be to add an __init__.py to twisted/plugins or to list all packages explicitly instead of using "find_packages()".

HACK yukyuk:~/playground/nevow/nevow-install-2527/Nevow$ svn diff
Index: setup.py
===================================================================
--- setup.py    (revision 16895)
+++ setup.py    (working copy)
@@ -70,6 +70,9 @@
     from setuptools import setup, find_packages
 
     setupdict['packages'] = find_packages()
+    setupdict['packages'].append("twisted.plugins")
+
+    setupdict['install_requires'] = ["Twisted >= 2.4"]
 else:
     # No setuptools -- decide where the data files should go and explicitly list
     # the packages.

(In [16896]) apply patch from comment

refs #2527

Okay, this is a cleaner solution to the old problem (no twisted/plugins/nevow_widget) and also fixes the new problem (no nevow/test/test_package):

Index: setup.py
===================================================================
--- setup.py    (revision 16896)
+++ setup.py    (working copy)
@@ -69,8 +69,7 @@
     from setuptools import setup, find_packages
 
     setupdict['packages'] = find_packages()
-    setupdict['packages'].append('twisted.plugins')
-    setupdict['install_requires'] = ["Twisted >= 2.4"]
+    setupdict['include_package_data'] = True
 else:
     # No setuptools -- decide where the data files should go and explicitly list
     # the packages.

Here is the setuptools doc about "include_package_data":

""" include_package_data

If set to True, this tells setuptools to automatically include any data files it finds inside your package directories, that are either under CVS or Subversion control, or which are specified by your MANIFEST.in file. For more information, see the section below on Including Data Files.

"""

from http://peak.telecommunity.com/DevCenter/setuptools#new-and-changed-setup-keywords

This patch removes the explicit dependency on Twisted only because that is part of a separate ticket -- #2629 (Nevow doesn't declare its dependency on Twisted in a machine-parseable way).

Here is the most recent version of this patch, ported to trunk [16896]:

The current version [16898] passes the current tests http://buildbot.divmod.org/builders/linux32-py2.5-nevowinstall/builds/110 . Please review.

With setuptools installed, these warnings pop up now:

warning: no files found matching 'setup_egg.py'
warning: no files found matching 'setupcommon.py'
warning: no files found matching 'test.py'
warning: no files found matching '*.as' under directory 'doc'
warning: no files found matching '*.fla' under directory 'doc'
warning: no files found matching '*.gif' under directory 'doc'
warning: no files found matching '*.jpg' under directory 'doc'
warning: no files found matching '*.png' under directory 'doc'
warning: no files found matching '*.txt' under directory 'doc'
warning: no files found matching '*.ics' under directory 'doc'
warning: no files found matching '*.pdf' under directory 'doc'
warning: no files found matching 'examples/pastebin/data'
no previously-included directories found matching '*/.svn'
no previously-included directories found matching 'doc/html/*.html'

Does that mean there's some more install cruft we can delete?

Otherwise I think the branch is in good shape.

Testing whether I get mail when this ticket changes now that I added "zooko@zooko.com" and "zooko" to Cc:.

Hm. setup_egg.py and setupcommon.py were both removed. Ah! I see there is a MANIFEST and a MANIFEST.in that mention them.

If you do have setuptools installed, then all files under revision control are includes in distributions, so they don't really need to be specified in MANIFEST.in. But in order to retain the ability to build the same distributions without setuptools installed, we should make sure that all appropriate files are included in MANIFEST.in. I leave that job up to you, exarkun. In order to stop hearing these warnings that you mentioned, we should ensure that some inappropriate files are removed from MANIFEST.in. Clearly setup_egg.py and setupcommon.py should go. Not sure which test.py it is talking about, and I know very little about the other warning examples. So I guess I leave that job up to you as well.

Please let me know if you want me to go ahead and figure it out and/or guess, myself, or go ahead and tell me what to do but I will do the actual editing of MANIFEST.in and svn committing it to the branch. Please e-mail me directly at zooko@zooko.com to let me know that this ticket has changed (because #2698).

Thanks!

Hooray, exarkun committed [16990], and the tests on this branch passes:

http://buildbot.divmod.org/builders/linux32-py2.5-nevowinstall/builds/113

The tests on trunk fail:

http://buildbot.divmod.org/builders/linux32-py2.5-nevowinstall/builds/114

So it sounds like this branch should be merged.

adding keyword "review". I believe that this is the magic incantation to attract the attention of the great and mysterious exarkun.

Other tickets that are blocked -- at least in my mind -- on this ticket: #2629 (Nevow doesn't declare its dependency on Twisted in a machine-parseable way) and #2699 (build nevow without importing nevow).

(In [17016]) Branching to 'nevow-install-2527-2'

Definitely getting close. I set up a Python 2.4 builder to test Nevow installation. It looks good for the nevow-install-2527 branch, but I merged the branch forward to pick up anything new in trunk and there are some failures in the branch now. Results at http://buildbot.divmod.org/builders/linux32-py2.4-nevowinstall/builds/6 and http://buildbot.divmod.org/builders/linux32-py2.5-nevowinstall/builds/120

The failures seem to all relate to test_howtolisting, which I don't think is too surprising, as those tests do some unusual manipulation of the import path. Not sure what the best fix is here. There's already a ticket for this problem, #2713.

It may be the case that these failures don't need to prevent this branch from being merged now. They're not strictly related to setuptools compatibility. However, because the value of a builder with any failing tests is significantly lower than the value of a builder with no failing tests, I think it would be vastly preferable to get #2713 resolved ASAP, lest new failures accumulate that do impact setuptools compatibility without anyone's notice.

Assigning back to you for comment.

[17018] makes those tests pass from installed locations. I tested it locally with setuptools installed. I didn't test it without setuptools installed, but it is expected to be the same.

Um, assigning back to you for (hopefully) merging.

(In [17073]) Branching to 'nevow-install-2527-3'

(In [17075]) Merge nevow-install-2527-3

Author: zooko Reviewer: exarkun Fixes: #2527

Change Nevow's setup.py to use setuptools if setuptools is available. This has the user-facing consequence of installing Nevow as an egg if setuptools is available at installation time and of making Nevow installable using the `easy_install´ tool.