Monday, October 29, 2012

Adding Moodle 1.9 Block Restore to Moodle 2.3 - Part 4

At this point, I think I need to step back and try to understand how the conversion process is executed so that I can make some good decisions on how to structure the programming solution. So, I'm going to make one more run through the code to see what is happening. I'll warn you now, this post will be difficult to read, and attempts to document my search-and-reverse-engineer the code for 1.9 conversion.

I feel like if I could just figure out how the moodle1_question_bank_handler class is used I could be most of the way there... It seems like most of the work is done in this class's process_question function, but I can find no place in Moodle where this function is actually called, meaning it must be a constructed function in the Moodle code.

As an aside, what I mean by a "constructed function" is that the function name that is called is actually determined by logic at execution time and stored in variable names. This is a strong programming feature in PHP that is used in Moodle. It allows for function calls to be dynamically constructed from variable names. While a powerful programming tool, it can make reverse engineering very difficult.

Given this, it is likely that the act of registering the question bank handler in moodle1_converter::register_handler()causes some chain of events that allows process_question to be called somewhere!

Inside moodle1_converter::register_handler() is where it begins to get obscure, but it looks like the handler information gets stored in the moodle1_converter::pathelements variable as well as within the moodle1_converter::xmlprocessor object. Searching for use of the pathelements variable, I find it processed in the moodle1_converter::process_chunk(), path_start_reached() and path_end_reached() functions. All of these functions have specific code sections set up to handle blocks in a similar way to modules.

Likely, the function call is created in process_chunk by the use of the get_processing_object() and the get_processing_method() functions. The only place I can find a likely definition of the get_processing_method() function is in the convert_path class. And convert_path objects are loaded and returned by the get_paths function in every handler including the moodle1_question_bank_handler.

And there it is! The handlers' get_paths function creates new instances of convert_path (located in "/backup/converter/moodle1/lib.php"), loading a name and a path argument. The name argument of this function is used to construct the "process_[name]" function. This in fact happens in the convert_path constructor with the line:
$this->set_processing_method('process_' . $name);
Also noted, is that this function definition is a default definition, meaning any implementing class can change this. This may come in handy when I am adding all blocks to be converted.

Looking specifically at the question handler again, it instantiates a convert_path object, passing the name "question" to it, which is used to define the processing method as "process_question". This function is then returned in the get_processing_method function call made in the moodle1_converter::process_chunk function call before dispatching the function itself. Now, I just need to determine what calls process_chunk.

Searching further, process_chunk() is called by moodle1_parser_processor::dispatch_chunk() which appears to be only called by the parent classes of this object.

Stopping to think, at this point, I probably don't have to worry about where the calls are made from. I can derive that, the act of loading convert_path objects with appropriate function call information will at some point, trigger their execution. So, I just need to jump back to the block handler code and redesign it so that it loads paths for all blocks. The act of loading functions for these will allow me to control the conversions! Eureka!

I think I now know enough to design the programming structure for the block conversion.

Friday, October 12, 2012

Adding Moodle 1.9 Block Restore to Moodle 2.3 - Part 3

This is part three of my series on writing the Moodle 1.9 to 2.3 block restore conversion code for Moodle core.

Up to this point, all I've really done is figured out some of the basic operation of the Moodle 1.9 restore converter, and verified my findings using some new test code. This exploration has shown me that the heavy lifting I need to do is convert the data format from a 1.9 backup file to the data format used by Moodle 2. Essentially, I need to look at what data was stored in a 1.9 backup, compare it to what is in a 2.x format and plan for that transformation.

Looking at the backup XML first, in 1.9 I see:
<BLOCK>
  <ID>319066</ID>
  <NAME>participants</NAME>
  <PAGEID>26328</PAGEID>
  <PAGETYPE>course-view</PAGETYPE>
  <POSITION>l</POSITION>
  <WEIGHT>0</WEIGHT>
  <VISIBLE>1</VISIBLE>
  <CONFIGDATA>Tjs=</CONFIGDATA>
  <ROLES_OVERRIDES>
  </ROLES_OVERRIDES>
  <ROLES_ASSIGNMENTS>
  </ROLES_ASSIGNMENTS>
</BLOCK>
In 2.3, I see:
<block id="10" contextid="278" version="2012061700">
  <blockname>participants</blockname>
  <parentcontextid>16</parentcontextid>
  <showinsubcontexts>0</showinsubcontexts>
  <pagetypepattern>course-view-*</pagetypepattern>
  <subpagepattern>$@NULL@$</subpagepattern>
  <defaultregion>side-pre</defaultregion>
  <defaultweight>2</defaultweight>
  <configdata></configdata>
  <block_positions>
    <block_position id="1">
      <contextid>16</contextid>
      <pagetype>course-view-topics</pagetype>
      <subpage></subpage>
      <visible>1</visible>
      <region>side-post</region>
      <weight>1</weight>
    </block_position>
  </block_positions>
</block>
I need to analyze this data and structures and determine how I migrate the old to the new.

The easy ones (I believe) should be able to transfer "as is":
  • ID => id
  • NAME => blockname
  • CONFIGDATA => configdata
The others will require some transformation processing. I'll look at each one.

contextid - This would be the context instance id value of the block on the system it was backed up from. Since this is not kept in 1.9 backups, I'll need to create one. This is a common problem with all 1.9 converters, so there should be examples I can borrow.

version - This would be the version string of the block when it was backed up. This is not in the 1.9 backup information. I'll have to find what I can use in its place, possibly leaving it blank.

parentcontextid - This is likely the context instance id of the context the block is installed on. This is not part of the 1.9 backup, but the page id (PAGEID) is. I may be able to use it, since what is probably most important is consistency for any block on that page.

showinsubcontexts - This is a new feature of blocks previously not available in Moodle 1.9. I should be able to just default this to zero.

pagetypepattern - This is a new setting for blocks in M2. This looks to be similar to PAGETYPE, but is a setting that impacts what pages it will displayed on. This will require a bit of investigation.

subpagepattern - This is a new feature of blocks previously not available in Moodle 1.9. It looks like I should be able to default this to $@NULL@$.

defaultregion, defaultweight - These are other new settings for M2 and can probably be set to default values.

block_positions - The block positions structure's elements pagetype, visible, region and weight will be based on the PAGETYPE, VISIBLE, POSITION, and WEIGHT elements but will require some transforms. The other elements in there, id, contextid and subpage will require more investigation.

So, I have some idea of what I need to do with the block data. Now I need to plan for where that will happen (so I can play with code and see what I need to do).

There is already a moodle1_block_handler class defined in the "handlerlib.php" file. This has been created primarily to be extended by any block that needs it. Recall from my previous post, that all handlers get registered for use in the moodle1_handlers_factory::get_handlers() function. In this function, any block that provides a moodle1_block_handler instance also gets registered. I need to also register a handler for all blocks to do the generic work associated with every block.I'm thinking, this should all be part of the moodle1_block_handler class, such that every block is managed by functions in this class. And that class should be responsible for calling any specific block backup code provided by those blocks. I'm going to try designing this approach to see if it makes sense.

After I've bounced this idea of some of the developers in the tracker, I'll come back to this.

Thursday, October 11, 2012

Adding Moodle 1.9 Block Restore to Moodle 2.3 - Part 2

This is part two of my series on writing the Moodle 1.9 to 2.3 block restore conversion code for Moodle core.

When I finished part one, I had discovered that I could begin defining block restore handlers by creating a converter file and providing a minimum handler function. But, my handler was not yet providing the minimum amount of information to remove the warning from the restore logs.

Looking back at the document, I see that the get_paths() function needs to return an array for all XML elements of the restore. My "participants" block needs a path "/MOODLE_BACKUP/COURSE/BLOCKS/BLOCK/PARTICIPANTS" defined. This can be confirmed by the warning in the log file. As an experiment, I change the function in "/blocks/participants/backup/moodle1/lib.php" to look like:
public function get_paths() {
    return array(
        new convert_path('participants', '/MOODLE_BACKUP/COURSE/BLOCKS/BLOCK/PARTICIPANTS'),
    );
}
I also note that the document indicates that one process_xxx() function must exist for every path defined in get_paths(). So I also add:
public function process_participants($data) {
}
Retrying my restore, and reviewing the restore logs, the warning is now gone for my "participants" block. I am definitely on the right track.

Next, I'm going to review the "Data structure transformations" section to see what I need to do.

This section shows that the get_paths() function also provides transformation information for data elements that have changed from 1.9 to 2.x. This documentation is specific to modules but does help understand what needs to be done for blocks. I do know that quite a bit of the block data structure changed in Moodle 2, so I will need to create a transformation. But, in general, the block data that changed is generic to all blocks. I don't think I should be creating general block data transformation code in a specific block's handler. I need to do more research.

At this point, I am going to do a bit more code searching, to see if I can figure out what is happening internally. I know that everything essentially starts in the "/backup/converter/moodle1/lib.php" file with the moodle1_converter class. Inside that class, I find the init() function, which adds the MOD and BLOCK plug-ins to the object. Much of this is done through calls to the moodle1_handlers_factory class in the "/backup/converter/moodle1/handlerlib.php" file. Specifically this code, brings in each block's conversion handler:
$handlers = array_merge($handlers, self::get_plugin_handlers('block', $converter));
...
protected static function get_plugin_handlers($type, moodle1_converter $converter) {
    global $CFG;

    $handlers = array();
    $plugins = get_plugin_list($type);
    foreach ($plugins as $name => $dir) {
        $handlerfile  = $dir . '/backup/moodle1/lib.php';
        $handlerclass = "moodle1_{$type}_{$name}_handler";
        if (!file_exists($handlerfile)) {
            continue;
        }
        require_once($handlerfile);

        if (!class_exists($handlerclass)) {
            throw new moodle1_convert_exception('missing_handler_class', $handlerclass);
        }
        $handlers[] = new $handlerclass($converter, $type, $name);
    }
    return $handlers;
}
What this is doing is looking at each block in the system, and seeing if each has a converter handler available, by looking for a "/blocks/[blockname]/backup/moodle1/lib.php" file, and then checking to see if it has a moodle1_block_[blockname]_handler class within it. If there is, then that handler gets registered for use in the conversion. If the file does not exist, then it moves on. If the file exists, but the class does not, then an error is thrown.

This code explains what I've seen in my restore tests. Originally, I was seeing the restore log indicating that none of the blocks had attached handlers. This is because there was no file with the appropriate class for any of the blocks. When I added that file and class to the "participants" block, I initially saw an error. The error occurred because the handler was registered, but the class was not providing the proper functions. Once the function was provided, I no longer saw the "no handler attached" message in the restore log, because my handler had been successfully registered in the code above.

I also notice that in the moodle1_handlers_factory::get_handlers() function, there are a number of handlers registered that are contained within the same file, using the code:
$handlers = array(
    new moodle1_root_handler($converter),
    new moodle1_info_handler($converter),
    new moodle1_course_header_handler($converter),
    new moodle1_course_outline_handler($converter),
    new moodle1_roles_definition_handler($converter),
    new moodle1_question_bank_handler($converter),
    new moodle1_scales_handler($converter),
    new moodle1_outcomes_handler($converter),
    new moodle1_gradebook_handler($converter),
);
I am now thinking, that I need to start with a more generic block handler, similar to one of the above handlers. Looking back at David's comment in the original tracker issue about following "a pattern similar to the one we used for Question bank conversion" leads me to believe this is the right track. Next, I will need to explore how questions are handled in the converter.


Adding Moodle 1.9 Block Restore to Moodle 2.3 - Part 1

I discovered recently, when trying to restore a Moodle 1.9 course into a Moodle 2.3 site, that the blocks that were part of the 1.9 course were not restored at all to the Moodle 2.3 course. This is a problem for me and my clients.

To look into the situation, I went to the Moodle tracker to see if  there was any information about this. I found MDL-32880.

After some communication through the issue, and through other development channels, I discovered that in fact that functionality had not made it into Moodle 2 yet. This led me to start looking for what it would take to get this functionality done, and if you look at the tracker issue, well, led me to be the one assigned to the task! Be careful when you offer to help.

Anyway, since I now have to figure out what to do, how the Moodle 1.0 to Moodle 2 conversion works and how the entire Moodle 2 restore framework works, I am inviting you along.

This series will document my efforts to build the Moodle 1.9 to Moodle 2 course block restore functionality. You should have a good working knowledge of PHP and Moodle code to be able to follow along. And, I since this is a work in progress, I cannot guarantee success. But you should at least learn what I do.

. . .

The first struggle I have come up against is understanding how Moodle 1.9 courses get identified and how the execution structure works to convert and restore them into Moodle 2. There are some hints in the tracker item, referring to handlers and restore log output. For example, there is reference to the following messages in the restore log when blocks are attempted:
[Wed Aug 29 15:25:32 2012] [warn] (moodle1) no handler attached /MOODLE_BACKUP/COURSE/BLOCKS/BLOCK/PARTICIPANTS
[Wed Aug 29 15:25:32 2012] [warn] (moodle1) no handler attached /MOODLE_BACKUP/COURSE/BLOCKS/BLOCK/ACTIVITY_MODULES
[Wed Aug 29 15:25:32 2012] [warn] (moodle1) no handler attached /MOODLE_BACKUP/COURSE/BLOCKS/BLOCK/SEARCH_FORUMS
[Wed Aug 29 15:25:32 2012] [warn] (moodle1) no handler attached /MOODLE_BACKUP/COURSE/BLOCKS/BLOCK/ADMIN
[Wed Aug 29 15:25:32 2012] [warn] (moodle1) no handler attached /MOODLE_BACKUP/COURSE/BLOCKS/BLOCK/COURSE_LIST
[Wed Aug 29 15:25:32 2012] [warn] (moodle1) no handler attached /MOODLE_BACKUP/COURSE/BLOCKS/BLOCK/NEWS_ITEMS
[Wed Aug 29 15:25:32 2012] [warn] (moodle1) no handler attached /MOODLE_BACKUP/COURSE/BLOCKS/BLOCK/CALENDAR_UPCOMING
[Wed Aug 29 15:25:32 2012] [warn] (moodle1) no handler attached /MOODLE_BACKUP/COURSE/BLOCKS/BLOCK/RECENT_ACTIVITY
As well, there is a recommendation to look at the "/backup/converter/moodle1/handlerlib.php" file.

From this, I learn three things. First, the warning message comes from the process_chunk function in "/backup/converter/moodle1/lib.php". Second, there is an empty, defined block handler in "/backup/converter/moodle1/handlerlib.php" called moodle1_block_handler. Third, I still have no idea how any of this works together.

After a few more questions and a few more code searches, I have some more useful information. First, there is an architectural document called "Backup 1.9 conversion for developers" that begins to describe the process of conversion and the execution flow of the restore process. Second the log information for the restore is in the "moodledata" directory located in "/temp/backup" in files called "{restore-controller-id}.log". This will help me to run some tests and determine if I am on the correct track.

Because there isn't always a good document to follow when building something like this, I try to use what's there to determine what is needed. The log file and the code in process_chunk and moodle1_block_handler are telling me that the Moodle 1.9 to 2 restore conversion process already is set up to try and perform block conversion. I want to find out how much is already there for me.

Looking at the document I referred to, and the code used by course modules, I can see that any block will probably need a "backup/moodle1/lib.php" file to define the restore conversion code. And looking at the warnings in the restore log file, it looks like that file needs to have a block restore handler. And, looking at the way modules are restored, I believe that handler needs to extend the moodle1_block_handler class and add the block name to the extended class like moodle1_block_[blockname]_handler.

Armed with that, I'm going to experiment using the "Participants" block. I create a new "/blocks/participants/backup/moodle1/lib.php" file as follows:
<?php

/**
 * Provides support for the conversion of moodle1 backup to the moodle2 format
 *
 * @package    block_participants
 * @copyright  2012 Mike Churchward (mike@remote-learner.net)
 * @license    http://www.gnu.org/copyleft/gpl.html GNU GPL v3 or later
 */

defined('MOODLE_INTERNAL') || die();

/**
 * Block Participants conversion handler
 */
class moodle1_block_participants_handler extends moodle1_block_handler {

}
Now, I am going to retry my Moodle 1.9 restore, and see what happens. If I get it right, the warning for that block should not appear in the restore log. Re-running the restore operation, I am stopped with the error:
Fatal error: Call to undefined method moodle1_block_participants_handler::get_paths() in /home/www/moodlehq.git/backup/converter/moodle1/lib.php on line 138
Believe it or not, this is reassuring. My addition to this block changed the way the restore operation worked. This means that there are definitely hooks already in the 1.9 to 2 restore conversion for blocks that are called when present. In this case, my new class was instantiated, executed and found wanting. But, it was executed!

Looking at the document I referred to, I see that it mentions that all module handlers must define the function "get_paths". This is also the function noted as missing in the error message I received. I'm going to add that function, but have it return an empty array for now, like:
public function get_paths() {
    return array(
    );
}
Retrying my restore operation, yields no errors. No blocks either; but I didn't expect any yet. And, unfortunately, I am still getting the "[warn] (moodle1) no handler attached /MOODLE_BACKUP/COURSE/BLOCKS/BLOCK/PARTICIPANTS" warning in the restore log. Still, I am pretty sure I now know where my work needs to begin. I will look more deeply into what the minimum handler requirements are, and see where I can go from there.

I'll pick this up in my next post.