Things accomplished at the hackathon

Aaron Crane (‎arc‎)

  • Perl 5 Configure:
    • Learned from Tux how to manage changes to Perl 5's Configure script, its metaconfig units, and porting changes both up- and downstream
    • Wrote and port several Configure changes to metaconfig units
    • And we synchronised the Configure script in 5.24.0-to-be with metaconfig!
  • Dealt with some Perl 5-related tickets: perl#127232, cpan#100183
  • Worked on new non-blessed-reference functions for Ref::Util
  • Code archaeology for Perl 5 SvRXOK macro, ready for backporting to Devel::PPPort
  • Helped jberger with a __SUB__ polyfill for older Perls, beyond what Sub::Current manages
  • Investigated how to let Devel::Cover see code outside subroutines in a required module (though not yet successfully)
  • Blogged!

Abe Timmerman

Barbie

  • Fixed the registrations for the CPAN Testers Wiki. Not sure how long this was broken, but thanks to Oriol Soriano Vila (‎Uree‎) for alerting me.
  • CPAN Testers Reports is now running on Fastly (http://fastly.com), allowing us to caching some of the pages, and reduce the load on the webserver when trying to recreate reasonably static pages. Also means the routing for anyone viewing the site outside of Europe is going to reduce page load times too. Thanks to Leo Lapworth (‎ranguard‎) for showing me the ropes.
  • Fixed some bugs in the Reports Mailer, refreshed the tests and test data, and tidied up the notifications.
  • Fixed the individual sending of reports via Reports Mailer, as this has been failing for some time. Due to the fact that we switch the DB storage to use Sereal, but didn't update the daily script to deserialize correctly before sending the individual reports. Note this had no effect on the summary reports.
  • For a long running bug with the Summary panel, which turns out has also been affecting MetaCPAN. After the server crash last year, turns out the base db schema was missing a change to one of the tables (part of the summary chain) and wasn't getting updated. Currently rebuilding, so expect to have the summaries and release data feeds all working again soon.
  • Continued to hand over the final keys to Doug Bell (PREACTION), who is now carrying the torch for CPAN Testers. I feel immensely confident that I have left the keys in capable hands, and with the ideas Doug has already shown me, I expect bigger and better things for CPAN Testers' future. Please look after him :)
  • Fixed a few bugs in other distributions, a couple related to CPAN Testers.
  • Cleaned up some of the CPAN Testers family website templates.
  • In the coming weeks, will be collating all the website tests I run prior to updating the CPAN Testers family websites, and handing over to Doug for his new development environment for CPAN Testers. This will hopefully enable easier access to anyone wanting to help fix problems on the websites and backends in the future.
  • Joined discussions for The Perl River, the (re)naming of the QAH and the future of CPAN Testers.

H.Merijn Brand (‎Tux‎)

  • Backported all open issues from perl5 blead to metaconfig, taking Aaron by the hand explaining all the steps and possible issues that can arise.
  • Merged all Configure changes into a new generated set for perl5 and merged that back so Ricardo Signes (‎rjbs‎) could merge that into 5.24.0-RC2 :) \o/
  • Tested the most recent Test::Smoke after discussions with Abe Timmerman (‎abeltje‎) on the just installed 5.24.0-RC2. The report was fine and entered the database which showed fine on the new website for smoke results
  • Started merging units from dist/git to dist/U so changes to U/modified can be identified easier
  • Removed U/modified units that have no change anymore compared to dist/git (after upstream merged suggested changes), to ease maint
  • Updated a few U/modified units based on improvements in dist/git (166 units to go)
  • Updated the documentation for metaconfig with all findings from doing the process on a Mac.
  • Attended the discussion about the river quality of CPAN and granted co-maint to several people to Release::Checklist
  • Attended the discussion about the implications of merging Test2 into perl5 core.
  • Installed most recent Test2 on laptop and tested that against all my modules
  • Fixed an issue in DBD::CSV that was actually in SQL::Statement and DBI::SQL::Nano (patches applied to both)
  • Installed the newest DBI/devel on my laptop and tested that against all installed databases (Pg, SQLite, mysql, CSV) and with an interface layer (Tie::Hash::DBD)
  • Sat with Tadeusz Sośnierz (‎tadzik‎) to get rakudo build bleading edge perl6 (moar & nqp) so my timings make more sense
  • Sat with user:lizmat to optimize Text::CSV for perl6 and then fixed a problem with aliasing multi-methods (ticket filed)
  • Discussed a problem in Pod::Simple with Neil Bowers (‎NEILB‎) and BinGOs
  • Released a new version of Spreadsheet::Read after analyzing Slaven's test failures
  • Tested most recen version of Devel::Cover on NetBSD for Paul Johnson (‎pjcj‎)
  • Split off System::Info from Test::Smoke for use in CPANTESTERS reporters. It's on github now
  • Digging into Darwin failures for Test::Smoke
  • Testing 5.24.0-RC2 on HP-UX after a failure report from jhi. WFM

bulk 88

Thursday

  • 2 patches were written to deal with perls with XSConfig installed, where there is no easy way to modify the faux-read-only %Config hash unlike with PP Config. Disclaimer, I'm the maintainer of XSConfig. cperl just forks T::H to make it pass with XSConfig, but that isn't abstracted correctly. The first patch which used the CPAN Mock::Config module was decided to be unacceptable after discussion with leont since it added an upstream dep to T::H, and T::H has a no deps other than core policy, since its a core bundled module by p5p. The second patch created a global package var which stored the info needed and could be runtime changed/mocked for testing, instead of needing to runtime change the RO %Config hash.
  • A heisenbug in proverun.t took most of Thursday to diagnose. After adding my WIP async code, proverun.t always failed under T::H when doing a "make test" yet never failed when running it alone in console. Stepping, print debugging, or adding sleeps also made it go away. You wouldn't think it, since normally the test harness/tap parser's module stack consists of installed modules, not possibly buggy non-CPAN or CPAN uninstalled modules, but when you do a "make test" on T::H, the uninstalled T::H is the harness that tests the uninstalled T::H. So instead of the "server" being stable version of the framework, and the "client" being the unstable version, both are the unstable version. This makes you wonder where is the bug. The bug was eventually discovered to be a override sub ref being assigned to a glob in a child class, and the glob/sub in that child class never existed, instead the meth was inherited from a base class but the dev made a mistake in thinking it was implemented in the child class when it wasn't. The dev even ironically commented in the source code on the side effects of the heisenbug and didn't know why it was happened from when he wrote the code, but in practice the bug wasn't seen in real life, so the comment was wrong, until I replaced the IO layer and the symptoms that dev described in the comment happen in practice. Because the STDOUT of proverun.t can be interrogated with perl's -t operator, and T::H did class selection based on the result of -t operator, the override sub was never called under the async IO layer, but was called when run under console. The override method was supposed to suppress T::H in a .t from writing to STDOUT. STDOUT should not be getting written to by T::H in the .t, since the .t was Test::More testing Test::Harness inside the same proc, and Test::Harness was corrupting the TAP stream by writing to STDOUT and causing corruption that Test::Harness in a outer proc (the make test proc) was reading in and thus failing. make_tool->seperate_proc->test::harness for real->seperate_proc->.t->test::more->test::harness was the app stack for proverun.t.
  • In some places inside T::H, perl exceptions from other parts of T::H are trapped and stored in objects, but some/most/many .t'es in T::H's test suite, never fetch the error/exception data from objects, ever. So serious errors (dies that I added during development) that should break the T::H .ts were being swallowed without a trace by the .t. This couldn't be found without extensive single stepping of T::H's code and made adding die()s seemingly never execute. I didn't suspect exception swallowing, until I as an experiment did a unpack('P1', "\x01\x00\x00\x00") to force the perl proc to SEGV instead of die() since I started to not trust the die()s I wrote. When I discovered the SEGV executed, but the die() in the next statement right below didn't seem to execute, I knew exceptions were being blindly swallowed and to start single stepping.
  • Since speed is the only thing I care about, in discussion of replacing Test::Builder, since I never use anything fancier than Test::More, I benchmarked Test2::Bundle::More vs legacy Test::More. Found Test2::Bundle::More 45% faster than Test::More in execution time of sub ok(). Reported findings to Exodist. Told him my positive feelings and did not participate in any other Test2 discussion that day, or on retrospect, ever again during QAH since I dont have much skin in T::B vs T2 debate.
  • Met Mithaldu offline for first time ever. A fellow Win32 Perl user I've known for a long time online.

Friday

  • Tap::Parser::Iterator::Process was split into a Unix (really generic/any OS) and Windows class. Design complexity was that multiple CPAN modules do Tap::Parser::Iterator::Process->new() and the implementation of a T::H process had to goto the correct implementation. For stability reasons and because the actual async IO on Win32 subs are a separate XS module on CPAN, the Unix process class continues to work on Win32 in serial testing mode, just like it was before my code. T::H has its own object class called TAP::Object and it was never designed for runtime dynamic class selection where a creating a base class instance actually creates a obj of a derived type, not the type requested. Infinite recursion and perl stack exhaustion happened on the first try as *Tap::Parser::Iterator::Process->new(Tap::Parser::Iterator::Process->new(Tap::Parser::Iterator::Process->new()))* happened. Since T::H has its own object system, that uses "_initialize" methods instead of perl standard "new" methods, the 2nd try called an empty stub/default "_initialize" in TAP::Object instead of the intended Tap::Parser::Iterator::Process::* namespace "_initialize". The prototype of _initialize does not allow changing the classname unlike perl standard new methods. Use of a stepping debugger proved essential to writing this code/patch. Also T::H extensively uses closures and anon subs and sub factories to implement C++ private object members in Perl, which is unperl-like and this had to be reversed and minimized in some places.
  • A Win32 IO Completion Port implementation of Tap::Parser::Iterator::Process was added. Difficulties included that "reads" must be reattempted, if no newline was found in the data that was returned by an async read, and since Windows IO Completion are a FIFO queue in which a read packet for ANY .t proc can be returned in a unpredictable order, yet a Tap::Parser::Iterator::Process->next() (really a read/fetch/get method) must return a line on a specific fixed .t proc, a system had to be invented to emulate sync blocking IO from 1 proc ontop of a queue where IO was poped from ANY proc. The emulation was done by a while loop that runs until 1 read or 1 line of TAP is available from the specific .t proc, and reads and lines of TAP from other .t procs are parsed and queued in the Iterator objects for those other .t procs. Eventually a ->next() will be called on those other Iterator objects which will non-blocking drain the parsed TAP lines. Although this design can cause the parsed TAP line buffers to grow very large and grow unequally nothing can really be done about when round robin queue must be emulated ontop of a queue where the largest TAP emitter .t proc gets the largest amount of timeslices. Remember that .t files all emit a different TAP tests per second than other .t files due to inherent underlying work, and OS thread scheduler, and number of CPU cores differences. Round robin is a bad idea for .t files, since .t files that are sleeping should get 0 time slices from T::H, but Multiplexer.pm without a working IO::Select requires that round robin (iterating a perl array from index 0 going up) work.
  • The first implementation of introducing a new IO layer, specific for Win32 (or any OS) into T::H had the random proc queue, but since only 1 child proc .t was ever started by T::H for the majority of T::H's test suite, no sorting of read events into respectic procs needed to be done, since there was only 1 running child proc at a time. A PP assert made sure the opaque perl ref sent through the async queue and poped off the queue when the read event is delivered was identical to $self. The assert compared the PID stored in the opaque obj and self obj (both are of type Tap::Parser::Iterator::Process). The assert 1 out of 10 runs failed in multiplexer t since by chance the order of the events coming off the queue when there were 3 procs running happened to be in the order the 3 procs were originally launches by multiplexer.t. Increasing the amount of child procs from 3 to 6 made the events be almost always 10 out of 10 times, interleaves between different procs rather than by chance all the events coming from 1 proc. This helped to diagnose the race condition and the the assert was removed when the previous bullet point on this page was implemented by me.

Saturday

  • Fixed a memory leak in Tap::Parser::Iterator::Process::Windows. The leak was caused by me not including a statement that was in Tap::Parser::Iterator::Process::Unix which broke a circular reference, where $self of Tap::Parser::Iterator::Process::* was closed over in next() method, since the next() method prototype had no arguments, not even $self. @_ was designed to be empty on entry to the method since T::H's last major dev work in ~2007. Changing the prototype of next method to make it sane should be done but was not tried due to high risk of breaking other parts of T::H's code and tests, and possibly compat CPAN addon modules to T::H. The memory leak was discovered by writing a temporary DESTROY method and breakpointing it. Bizarrely and unperl-like, there isn't a single DESTROY method in T::H. T::H instead uses circular references and C++ style explicit memory management with *_finish* methods to break circular references.
  • Added an Win32 async compatible IO loop to TAP::Parser::Multiplexer. Complexity includes that T::H is a pull architecture as leont describes in https://github.com/Perl-Toolchain-Gang/Test-Harness/issues/30 . I hacked on a push option to Tap::Parser::Iterator::Process::Windows, which is implemented by a combination of, "if set to push mode, forbid pulls (blocking IO AKA hangs)" and pushing data off the Win OS pipe handle into Iterator objects causes later possibly blocking "read" methods, to be guaranteed non-blocking, as the data is already queued in PP space, which matches the existing Multiplexer architecture. The push option I added adds a POC that a more efficient POSIX event loop than IO::Select (aio/epoll/kqueue) could be added in the future to T::H. Other difficulties include analyzing what seem to be or are circular references and convincing oneself that they are not bugs but by design. I discussed a number of designs with leont that we both rejected for various reasons. One rejected redesign of Multiplexer were weak references (not used at all by T::H previously and not used currently after my improvements) and callbacks from the Iterator object back to the Multiplexer object in a outer stack frame. Since non-parallel T::H harness test runs use _aggregate_single and not _aggregate_parallel, there may or may not be a Multiplexer object in the callstack at any random call into an Iterator object. Using caller() or CPAN modules that use basically perl debugging facilities to walk the callstack, to look for a Multiplexer object to notify which Iterator object/Parser object was ready to read was also rejected.
  • went to cpan river meeting

Christian Walde

Doug Bell

Wendy van Dijk

  • Went shopping every day, sometimes twice a day, for fruit, vegetables, cookies, nuts, soda, tea and other things
  • Spent several hours every day preparing the fruit and vegetables, and cleaning after the other people
  • Attended three meetings (Test2 & Test::Builder, River of CPAN, Test2), took notes, took a bit part in some discussions, and transcribed the notes and sending the texts to Chad, Neil and Ricardo.
  • Succeeded in wasting many people's time with silliness and booze.

Karen Etheridge (‎Ether‎)

  • fixed Dist::Zilla::Plugin::DynamicPrereqs's injection of Makefile.PL snippet using ExtUtils::HasCompiler for the can_xs() sub, and re-released B::Hooks::EndOfScope with the fix
  • released a new Moose (2.1705-TRIAL) with a similar change to use ExtUtils::HasCompiler in its Makefile.PL for XS sanity checking
  • discussed the old JSON::PP bug involving parsing unicode META.json (https://github.com/makamaka/JSON-PP/pull/9) and the injection of a JSON::PP "suggests" prereq via Dist::Zilla::Plugin::Git::Contributors with Ricardo Signes (‎rjbs‎) more; this led to a change in Dist::Zilla where it now creates pure-ascii META.json files, and a similar change to come in ExtUtils::MakeMaker for MYMETA.json generation
  • wrote a blog post: http://blogs.perl.org/users/ether/2016/04/its-earth-day---time-to-clean-up-cpan.html
  • released Olaf Alders (‎oalders‎)'s MooseX::Getopt fix for init_args (thanks Olaf!)
  • started to fix logic in Dist::Zilla::Plugin::MakeMaker regarding its choice of minimum ExtUtils::MakeMaker versions depending on the presence of version ranges in prereqs and added test coverage; shelved this work for now as it requires more refactoring than I have time for at the event
  • discussed the application of module permissions on new uploads to PAUSE with user:andk, user:bwfg etc and proposed changes to handing of x_authority metadata (similar to the semantics behind the prototype x_authority_from_module, see Dist::Zilla::Plugin::AuthorityFromModule)
  • (attempted to) reproduce various issues in cpanm-reporter with user:garu and dogfooded his fixes
  • CPAN river discussion with Neil Bowers (‎NEILB‎) and many others
  • review of Software::License issues and support policy with Ricardo Signes (‎rjbs‎), Kenichi Ishigaki (‎charsbar‎), Neil Bowers (‎NEILB‎) and others
  • various discussions of Test2 and upcoming releases (see also http://blogs.perl.org/users/chad_exodist_granum/2016/04/test2testbuilder-update-from-the-qah.html)
  • witnessed the birth Dist::Zilla 6! ..and a made few plugin releases in the wake thereof to fix up new issues thus revealed, or clean up code that is now possible
  • spent significant time examining outstanding Module::Metadata issues (especially regressions since the last stable release), reviewed previous unfinished work fixing these, writing additional tests.. discovered that both the documentation and the actual code surrounding the 'name()' interface are (IMHO) wrong and need significant rethinking and rework, but this is out of scope for the time available at the QAH, so I shelved this for now and went back to fixing the regressions, which I believe I have done now with the 1.000032-TRIAL release
  • (not yet finished - hopefully Sunday) review, discuss and merge outstanding Sub::Name pull requests, involving handling of unicode or binary symbols

pete

  • Tests for PAUSE web-front-end
  • Travis job for PAUSE

Chad 'Exodist' Granum

  • Test2 Discussion
  • Merge Test2 into Test-Simple
  • X-Breaks for Test2
  • Fix some bugs in Test2 found at QAH by testers
  • Unit testing for Test2::Harness
  • Another Test2 discussion
  • Benchmark perl test suite under Test2 (direct, not via Test::Builder)
  • Make Test2::Harness work in windows (Thanks Mithaldu!)
  • Find and debug shm issue in Test2 (Thanks Eserte!)

mst

  • Added new env vars to select JSON backends for Parse::CPAN::Meta so it can use Cpanel::JSON::XS, Mojo::JSON, etc. (currently shipped as devrel)
  • Wrangled puppet and postgresql replication for metacpan, mostly ready though certain parts of the puppet module are un(der)documented so not quite finished reverse engineering it all into puppet configs
  • Volunteered as resident 'blogs.perl.org anti-drama llama' so other people could do more useful things than get into arguments
  • Helped a bunch of people crowbar a bunch of things in infrastructure and systems-ish stuff, plus being fresh eyes for debugging a few times

Steffen Schwigon (‎renormalist‎)

  • Benchmark::Perl::Formance
    • Finetuned my benchmark bundle "perlstone2015" which I started last year.
    • Extended my ~300 Perl installations (of versions since 5.8.9, +/- threaded, +/-64bit) with the 5.23.* series and 5.24.0.
    • Updated all those Perls to use current same set of CPAN dependencies.
    • Ran the "fastmode" variants (reduced runtime) over most of these Perls.
    • Created new charts and a dashboard
    • First conclusions:
      • reported micro optimizations like the "small blocks" entry/exit improvement are visible
      • algorithmic benchmarks like binarytrees, fannkuch, fasta, nbody, mandelbrot became remarkably faster during 5.23.*
      • others like regexdna and spectralnorm became already faster with 5.20, and could keep that speed
      • some regex micro benchmarks run faster
      • however, the regex engine micro benchmarks generally became slower since 5.18, but at least kept stable at that level until 5.24.0.
    • *CAVEAT* - I could only run the scaled-down "fastmode" benchmarks. I will run the heavier-weight benchmarks throughout the next weeks.
  • Perl and CPAN::Reporter on L4 Linux
    • Built Perl 5.22.1 and enabled CPAN::Reporter on an Ubuntu 16.04 system running an "L4 Linux" kernel 4.4.
    • L4 Linux is a Linux variant running on the L4Re micro-kernel.
  • Co-maintaining modules
    • merged github changes from Lance Kinley for Net::SSH::Perl new ciphers, key exchange algorithms and more; released version 2.01 to CPAN, see
    • En passant also uploaded Lance Kinley's Crypt::OpenBSD::Blowfish to CPAN.

Slaven Rezić (SREZIC)

Olivier Mengué (‎dolmen‎)

Day 1

  • Dist::Zilla
    • New PrereqsScanner role
      • I find AutoPrereqs too slow. For an long time I had an idea about a replacement for AutoPrereqs power: scan only the dist source on release to check that dependencies detected mach dependencies that have been declared manually in another way (for example with a cpanfile)
      • So with Philippe Bruhat (‎BooK‎), refactored Dist::Zilla::Plugin::AutoPrereqs to extract Dist::Zilla::Role::PrereqsScanner to allow reuse by others scanners (Dist::Zilla::Plugin::AutoPrereqs::Fast) https://github.com/rjbs/Dist-Zilla/pull/532
      • Refactored Dist::Zilla::Plugin::AutoPrereqs::Fast to use that role. I still have to push the pull request
      • Still have to implement my initial idea though :(
      • The PrereqsScanner work made me had a look at dzil finder roles and I sent a patch https://github.com/rjbs/Dist-Zilla/pull/536
    • A month ago I implemented major optimisations in Data::OptList (see https://github.com/rjbs/Data-OptList/pull/1 ), so I sent a pull request to add a dependency on that version: https://github.com/rjbs/Dist-Zilla/pull/533
  • Pod::Spell
    • Merged patches
    • Made a new release
    • Had a look at the river below Pod::Spell. We know that we lack tools for that at the moment. But I got information from the work in progress at that QAH to improve that.
  • Test2/Test::Builder discussion

day 2

  • Test2
    • Installed the dev release of the new Test::Builder based on Test2
    • Started to upgrade all my local modules using that new test toolchain, with the help of App::cpanoutdated. I have CPAN::Reporter enabled, so the results are fed to CPANTesters.org
    • No failures reported. Great!

Olivier Mengué (‎dolmen‎) Days 2, 3, 4

  • To be continued for reports of days 2, 3, 4...

You

version 36 saved on 25/04/16 11:29 by Olivier Mengué (‎dolmen‎)

Home | Tags | Recent changes | History

Sponsors

Donate


FastMail

ZipRecruiter

ActiveState

OpusVL

STRATO

SureVOIP

CV-Library

Infinity Interactive

Perl Careers

MongoDB

think project!

DreamHost

Campus Explorer

The Perl 6 Community