Achievements

Testing Coverage

At the 2013 Perl QA Hackathon in Lancaster, England, testing coverage-related issues were handled by several participants, including the following:

Paul Johnson (pjcj)
Jim Keenan (kid51)
Abe Timmerman (abeltje)
Dinis Rebolo (drebolo)

... with remote assistance from Thomas Klausner (domm). The work had two major focuses:

1. cpancover: Improving reporting of testing coverage data on major CPAN modules, expanding on pjcj's cpancover.com.

2. perl5cover: Creating a mechanism to run coverage analysis over the C source code and Perl libraries in the Perl 5 core distribution and to make that analysis available.

cpancover

The purpose of the hackathon's work with respect to cpancover was to integrate cpancover with metacpan -- specifically, to place a link on a distribution's metacpan page pointing to the appropriate page in cpancover.

perl5cover

perl5cover will have two major aspects: one centralized, one distributed.

First, a centralized server will run a coverage report on the Perl 5 core distribution shortly after each release of a major (5.18.0) or minor (5.17.11) version of Perl. The coverage will use 'gcov' to report on the Perl 5 source code written in C and Devel::Cover-based tools for the Perl libraries (.pm files) which ship with the core distribution. At any point in time, a limited number of recent releases of major and minor versions will be displayed as web pages. However, plain-text versions of the coverage summaries will be retained indefinitely and be available for download.

Second, we will make available, probably in the form of a CPAN module, instructions showing how to run a coverage analysis on the core distribution at any commit.

If you are obsessed with how well covered the code in blead is at any moment in time, the second approach will be the better approach for you -- but you will have responsibility for maintaining the infrastructure.

If however, your objective is to identify inadequately tested parts of the Perl 5 core distribution and contribute appropriate, new tests to the test suite, then the first approach will be better. Our philosophy will be "KISS"; we'll try to get a basic, not-difficult-to-maintain infrastructure up soon. We'll worry about adding bells and whistles later.

This work will build on domm's App::ArchiveDevelCover, but we will probably end up creating one or more new CPAN distributions to achieve this. The hackathon saw a lot of work as well on the creation of a stand-alone program to run coverage of the core.

DBI::Test

DBI::Test is a new test suite controller and bundle to allow DBD's and other DBI related modules to test their spec compatibility. DBI::Test is to be run to test the DBI API from both sides (DBI and DBD).

Jens Rehsack (‎Sno‎), H.Merijn Brand (‎Tux‎), ribasushi +1 and joakim brainstormed and work on different parts. On the way back, Jens Rehsack (‎Sno‎) and Philippe Bruhat (‎BooK‎) discussed about Test::Database and decided to work together (merge some code and reponsibility from Test::Database and improve Test::Database to run in DBI::Test framework).

Jens Rehsack (‎Sno‎) will start working on the DBI::Mock part to allow DBI::Test can run self-tests without having DBI installed. Additionally, he's working on the setup and test population task in DBI::Test.

joakim works on the test modules from common basic ones up to complex ones later, when an initial setup is running. ribasushi +1 will function as primary mentor for Joakim's work while Jens Rehsack (‎Sno‎) is co-mentor.

H.Merijn Brand (‎Tux‎) doesn't do technical work in DBI::Test to keep a clear eye from DBI spec/API perspective.

The team will try to get as many DBD authors involved as possible and deal with feedback as soon as possible in order not to divert into a module that won't be used. We hope that this effort will show deficiencies on both ends.

Perl5 toolchain consensus

Each day a brainstorming meeting for 2-4 hours to discuss toolchain issues and find a community consensus for them.

Day one was for the toolchain and build process:

• 5.8.1 is the new minimum for toolchain Perl modules (was 5.6.1)
  • 5.8.1 gets us...
    • PerlIO
    • Modern (if buggy) Unicode
    • Modern (if buggy) threads
    • These modules are now available: Attribute::Handlers, Class::ISA, Devel::PPPort, Digest::MD5, Encode, List::Util, Scalar::Util, MIME::Base64, Memoize, Net::FTP, Storable, Test::More, Text::Balanced, Time::HiRes
  • If 5.8.1 proves too much work we reserve the right to move to 5.8.4 (Solaris default version).

• An explicit way to indicate to a module we would like a pure Perl build.
  • perl Makefile.PL PUREPERL_ONLY=1
  • perl Build.PL --pureperl-only
  • Existing PERL_MB_OPT and PERL_MM_OPT environment variables can be used to pass switches
  • Installation must be able to be moved to another machine/architecture
  • If it cannot provide a pure Perl install, the build must error
  • The problem of archlib vs perllib shadowing must be dealt with
    • Probably once we get the installed modules database

• The installed modules database is a good idea
  • It should be under @INC
  • It must deal with permissions environments such as local::lib
    • It should stack the same way modules stack in @INC
  • It must require only core modules
    • It may use non-core modules for non-required features
  • It is not a replacement for packaging systems
  • A large pile of possible uses for such a system were gathered
    • Installed tests
    • Dependency and reverse dependency information
    • Querying what is installed
    • Module uninstall
    • Clean upgrade (ie. uninstall before installing a new version)
    • Recompile/reinstall modules for a new major Perl version
    • Make packaging easier
    • Track how a module was installed (directly asked for or as a dependency)

Day two has been used to discuss meta spec:

• There is nothing pressing for a version 3 of the meta spec
• The meaning of conflicts is unclear but something like it is needed
  • The Debian Policy for Breaks and Conflicts should be examined
• optional_features vs recommends and suggests...
  • It was agreed we need optional_features
  • optional_features has poor tool support

On the third day testing issues (during the Perl module build process) were discussed:

• Automation will be split into 3 environment variables instead of one
  • AUTOMATED_TESTING is reduced to its initial meaning (this is a test bot)
  • NON_INTERACTIVE (newly introduced) meaning there is nobody at the keyboard
    • All prompted questions must use the default answer
  • EXTENDED_TESTING (newly introduced) meaning extended tests should be run
    • For example, slow tests
    • Extended tests do not include author tests (which were out of scope for this discussion), use AUTHOR_TESTING or RELEASE_TESTING
    • Extended tests should not increase the requirements of the test
  • See new modules Test::Is, Test::Settings and Test::DescribeMe that will help authors to easily follow this specification

• Installed module testing was discussed a long way with a lot of proposals to the author team from Norway. Everybody is excited about that approach.

• Should coordinate with the installed module database
• Must not say anything about how the tests are written
• New "make testinstall" and "Build testinstall" targets will be created in our build tools
• Should store the entire release archive at install time

New modules

• Bot::BasicBot::Pluggable::Module::MetaSyntactic - Philippe Bruhat (‎BooK‎)
• Test::Is - Olivier Mengué (‎dolmen‎)
• Test::Settings and Test::DescribeMe - Matthew Horsfall (‎alh‎)

New releases due vis-a-vis cooperation

Philippe Bruhat (‎BooK‎) worked with Christian Walde (‎Mithaldu‎) on getting System::Command (and therefore Git::Repository) to work on Win32. This led to the release of System::Command v1.100.

Perl 5+6 CPAN cooperation

A long discussion on Sunday about a Perl6 CPAN resulted in many agreements. The first thing everybody agreed on was that Perl6 (authors, module authors and users) needs the same CPAN access as Perl5. Technical details are open, but a fair way for now is uploading Perl5 modules and Perl6 modules into different subdirectories of the author home.

Further, everyone agreed that Perl is a language family whereby Perl5 and Perl6 are siblings.

Test::Smoke

This project for CORE smoking had some patches integrated that were lingered from several sources from over the past year. We also fixed (we hope) a double-encoding bug that sends logfiles as garbage (and fixed the wrong encoded logs in the database). We also integrated and tested new smoke ways to test smoke-me branches from git. All related projects have moved from svn to github by Abe Timmerman (‎abeltje‎).

Perl blogs

Aaron Crane (‎arc‎) and Philippe Bruhat (‎BooK‎) spent some time on setting up a use.perl.org mirror using WWW::UsePerl::Server, and adding support for redirecting historical URL to the proper content (stories, journals, comments).

Emscripten perl

David Leadbeater (‎dg‎) compiled microperl with emscripten, so that perl will work on a webpage. Still much work to be done (e.g. seeing if I can use Jess's cross-configure work to get a full perl). Demo at http://dg.cx/jsperl/.

Testing real dependencies

Jess Robinson (‎castaway‎) wrote everything up here: http://desert-island.me.uk/~castaway/blog/2013-04-Perl-QA-Day-3.html . See also github: http://github.com/castaway/Smoke-Real-Deps

Achievement unlocked: Asked Pumpking for commit bit in person (need to keep poking though ;)

Test::Builder 1.5

A small team worked through the roughly 100 of 700 Test-* distributions failing with Test::Builder 1.5 alpha but passing with the current 0.98 stable.

• Verify it's failing with 1.5 and passing with 0.98
• Verify it's because of a Test::Builder 1.5 change
• Categorize the failure type
• Patch either Test::Builder or the module as appropriate
• If the module runtime had to be patched, note it as a conflict.

We did the work primarily on an EtherPad and in the Test-More Github issue tracker by a small team of remote and local developers.

• Karen Etheridge (ether)
• James Mastros (theorbtwo)
• Colin Newell
• Daniel Perrett (pdl)
• Michael G Schwern (schwern)

Nearly all the modules got looked at, and a number patched.

Other accomplishments by the team...

• Determined that rolling back the TAP formatting changes in 1.5 will not significantly help
• Fixed Test::More stable and 1.5 alpha for 5.17.
• Released 0.98_04 and 1.5.0_6
• Improved the script for testing CPAN modules with stable vs alpha.
• Fixed the 1.5 alpha under taint mode
• Fixed the 1.5 alpha on Windows
• Misc documentation updates
• Got a new release of Test::Class working with 1.5

ExtUtils::MakeMaker stable release

Chris Williams (BingOS) stepped up to manage the next stable release of ExtUtils::MakeMaker to be in time for Perl 5.18.0 and released several alphas. They will be very well tested. :)

version 16 saved on 17/04/13 09:59 by Olivier Mengué (‎dolmen‎)

Home | Tags | Recent changes | History